How can we help you?

Best Practices for Splunk ITSI

As your ITSI deployment grows, maintenance can quickly become unwieldy. Adopting a few key best practices early can help improve performance, reduce administrative effort, and make your environment easier to scale.

We’ve navigated Content Packs, and we’ve unpacked ITSI entities. Now it's time we look at some practical techniques that can help you build a more maintainable and scalable ITSI environment.

What base searches are and why you should use them

In the simplest terms, you can think of a base search as a kind of saved search, specific to ITSI, that can be reused across multiple services. This means that the search only has to run once, limiting both complexity and resource usage.

However, for AC3's Cisco-Splunk Consultant Jade Bujeya, the biggest advantage is the ease of updates and scaling. When you create a new service, you can reuse logic you've already validated rather than starting from scratch. Updates only need to be made once, and documentation only needs to be written once. As ITSI deployments mature and become more complex, that kind of consistency and efficiency becomes invaluable.

An interesting hack from Jade

“Let’s say we have a customer who, for whatever reason, wants to be able to see CPU at the host, VM, Role, and Location levels. Or maybe the Data Base team wants it at the host level, the Application team wants to see it per VM, the Customer Service Team wants to see metrics by Role, and Management wants to see everything by Location.

Example Data

You’ve already written your search. You don’t want to make a bunch of copies with tiny variations, all of which you’re going to have to remember to update later, and all of which you’re going to have to keep track of and document.

Step 1: You take you original search and make it a saved search.

Step 2: You take your original base search, and make it call | savedsearch instead.

Step 3: Now you make your copies. They all call the same saved search, so you can have a single place for updates and doco, and they all Filter to Entities in Service using the host field.

Step 4: But, they each have different Entity Split Fields. One splits on host, another vm_name, another role, the last location. They’re all using the same data stream, but they’re grouping at different levels. Everyone gets what they need, without unduly adding to your administrative workload.”

Dashboarding with the itsi_summary and itsi_summary_metrics indexes

Jade stresses the value of dashboard with data from the ITSI indexes. “All of your data, that all of your KPIs, in all of your services produce, is stored and available here. Finding what you need can take a minute […] but it’s an amazing resource, and I encourage you to use it.”

Have more questions about ITSI?

Whether you're just getting started with ITSI or looking to optimise an existing deployment, AC3 can make a difference. Our Cisco-Splunk team has helped organisations design, implement, and mature ITSI environments of all sizes, and can help you get the most value from your investment.

Get in touch with AC3 to discuss your ITSI goals and next steps.