Splunk Architect

Job title: - Splunk Architect
Location: - 100% Remote Role
Duration: - 12 Months

Top 5 hard skills:
Splunk Architecture & Administration
Core Competencies:

• Design and maintain distributed Splunk deployments (search heads, indexers, forwarders, deployers)
• Manage indexer clustering and search head clustering for high availability
• Configure data inputs, parsing, and index management
• Implement role-based access control (RBAC) and authentication integration
• Performance tuning and capacity planning

Data Onboarding:

• Design and implement data onboarding strategies for diverse data sources
• Create and maintain props.conf and transforms.conf for data parsing and routing Develop source type definitions and field extractions
• Configure input specifications and monitor data quality post-onboarding o Establish data retention policies and index lifecycle management

Splunk HTTP Event Collector (HEC):

• Configure and manage HEC endpoints for REST API-based data ingestion
• Implement HEC tokens with appropriate permissions and index routing
• Troubleshoot HEC connectivity, authentication, and data formatting issues
• Scale HEC deployments for high-volume event ingestion
• Integrate cloud-native applications and serverless functions with HEC

Splunk DB Connect:

• Install, configure, and maintain DB Connect app across search heads
• Create database connections and manage JDBC drivers for various database types Design and schedule database inputs (rising column, batch, and tail inputs)
• Optimize SQL queries for performance and minimize database load
• Configure database identity management and credential security
• Troubleshoot connection issues, query timeouts, and data ingestion gaps

Relevance:

• Essential for maintaining platform health, scalability, ensuring data availability across the enterprise, and enabling seamless integration of diverse data sources into the Splunk ecosystem

AWS Infrastructure & Services Core Competencies:

• Deploy and manage EC2 instances for Splunk components with proper sizing Configure VPCs, security groups, NACLs, and networking for secure Splunk communication
• Implement EBS storage optimization and snapshot strategies for Splunk data Leverage S3 for SmartStore architecture and backup solutions
• Use AWS Systems Manager, CloudWatch, and Auto Scaling for monitoring and automation
• Relevance: Critical for cost-effective, secure, and resilient infrastructure supporting enterprise-scale log aggregation

Infrastructure as Code (IaC) & Automation Core Competencies:

• Terraform or CloudFormation for provisioning Splunk infrastructure
• Ansible, Puppet, or Chef for Splunk configuration management
• Python/Bash scripting for custom automation tasks
• CI/CD pipeline integration (Jenkins, GitLab CI, GitHub Actions)
• Version control with Git for infrastructure and configuration code
• Relevance: Enables repeatable deployments, reduces human error, and accelerates disaster recovery and scaling operations

Monitoring, Logging & Troubleshooting Core Competencies:

• Create Splunk monitoring dashboards and alerts for platform health
• Implement log forwarding strategies using universal/heavy forwarders  
• Troubleshoot data ingestion issues, search performance, and cluster health  Integrate AWS CloudWatch metrics with Splunk for unified monitoring
• Analyze Splunk internal logs (_internal, _introspection, _audit indexes)

Relevance:

• Ensures platform reliability, rapid incident response, and proactive identification of issues before they impact users

Security & Compliance Core Competencies:

• Implement encryption in-transit (SSL/TLS) and at-rest for Splunk data
• Configure AWS IAM roles and policies following least-privilege principles
• Ensure compliance with standards (PCI-DSS, HIPAA, SOC 2) for log data
• Implement backup and disaster recovery procedures
• Secure API access and credential management (AWS Secrets Manager, HashiCorp Vault)

Relevance:

• Protects sensitive log data, maintains audit trails, and ensures regulatory compliance in enterprise environments

Cribl Stream & Cribl Edge –

• Data Pipeline Optimization Cribl Stream (LogStream)

Competencies:

• Deploy and manage Cribl Stream architecture (Leader nodes, Worker nodes, Worker groups)
• Configure data sources and destinations for multi-platform routing (Splunk, S3, other SIEMs)
• Design and implement pipelines for data transformation, enrichment, and reduction Create routes and filters to optimize data flow and reduce ingestion costs
• Implement data sampling, aggregation, and redaction for compliance and cost savings
• Configure event breakers, parsers, and field extractions within Cribl
• Manage Cribl packs for pre-built data optimization solutions
• Integrate Cribl Stream with Splunk HEC and S3 for hybrid storage strategies  
• Monitor pipeline performance and troubleshoot data flow issues
• Implement GitOps workflows for Cribl configuration management Cribl Edge

Competencies:

• Deploy and manage Cribl Edge fleets for distributed edge data collection
• Configure Edge nodes as lightweight agents replacing traditional forwarders  Implement centralized management of Edge fleets through Cribl Cloud or Stream Leader
• Collect data from edge sources (logs, metrics, Windows events, syslog)
• Perform edge-side data processing to reduce bandwidth and central processing load Configure auto-discovery and dynamic data source management
• Manage Edge node updates, configuration versioning, and fleet-wide deployments Monitor Edge node health and connectivity across distributed environments  Implement edge-to-cloud data routing strategies for hybrid architectures

Incident Management & Service Request Support Core Competencies:
Incident Response:

• Triage and respond to platform incidents following ITIL or similar frameworks
• Diagnose and resolve P1/P2 incidents affecting Splunk availability or data ingestion
• Perform root cause analysis (RCA) and create post-incident reports
• Coordinate with cross-functional teams during major incidents
• Implement corrective and preventive actions to reduce incident recurrence
• Maintain on-call rotation and provide 24/7 platform support

 
Service Request Management:

• Process user access requests (account creation, role assignments, permission changes)
• Handle data onboarding requests for new applications and data sources
• Fulfill infrastructure change requests (index creation, retention policy updates, capacity expansion)
• Coordinate app installations and updates on search heads o Provision and configure new forwarders, HEC tokens, or DB Connect inputs
• Create custom dashboards and reports based on user requirements

 
Ticket Management & Communication:

• Utilize ticketing systems (ServiceNow, Jira Service Management, Remedy) Document troubleshooting steps and resolution procedures
• Maintain SLA compliance for incident response and service request fulfillment Communicate effectively with stakeholders on status updates and timelines
• Create and maintain knowledge base articles for common issues
• Escalate complex issues to vendors (Splunk Support, AWS Support) when necessary

Proactive Support:

• Conduct health checks and performance reviews o Identify trending issues and implement preventive measures
• Provide user training and guidance on Splunk best practices
• Participate in change advisory board (CAB) meetings for platform changes

Relevance:

• Ensures rapid resolution of platform issues, maintains high availability and user satisfaction, and provides structured support that aligns with enterprise IT service management practices
• Essential for maintaining operational excellence and meeting business-critical