msticpy - Compatibility release - Bokeh, VirusTotal, AzureCredentials

This is mainly a release to fix compatibility problems with some features deprecated in Bokeh 3.7

Also includes fixes for different behaviour of VTObject in VirusTotalV3 code - which messes up conversion to pandas dataframes.

Finally I've added some fixes for using AzureCliCredential and ManagedIdentityCredential. In cases where you are using AzureCLI authentication with a ManagedIdentity (such as in AzureML compute), the credential fails if you supply a TenantId when creating. The code now checks that it can obtain a token and, if not, falls back to creating the credential with no tenantId. Similarly, the default for ManagedIdentityCredential is now to create it only passing client_id (or None if this is not defined). It will fallback to previous behavior, if this fails. If that also fails, it will fall back to creating the credential with no parameters.

What's Changed

Compat fixes for Bokeh 3.7 by @ianhelle in https://github.com/microsoft/msticpy/pull/840
Avoiding vulnerable dependencies by @ianhelle in https://github.com/microsoft/msticpy/pull/843
- updating cryptography to >=43.0.1
- Add explicit dependencies for jinja2>=3.1.5 and tornado>=6.4.2 to avoid vulnerable versions

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.16.1...v2.16.2.post

- Python
Published by ianhelle about 1 year ago

msticpy - Maintenance release QueryEditor, PrismaCloudDriver

Highlights

This is largely a "fix-and-improve" release. Some important fixes to: - QueryEditor - Pagination and retry capability added to Prisma Cloud Driver - Dataclass issue

As of this release we are switching from GitHub actions publishing to an Azure DevOps pipeline (this is a Microsoft internal security requirement for enhancing supply chain security). It should not affect your enjoyment of this package :-)

What's Changed

Fix dataclass issue by @FlorianBracq in https://github.com/microsoft/msticpy/pull/833
Add typing for Defender by @FlorianBracq in https://github.com/microsoft/msticpy/pull/828
Edit process_cmd_line function template to accept list as parameter by @vx3r in https://github.com/microsoft/msticpy/pull/835
Fix to QueryEditor by @ianhelle in https://github.com/microsoft/msticpy/pull/836
Add pagination and retries based on load, support queries - PrismaCloudDriver by @raj-axe in https://github.com/microsoft/msticpy/pull/834
Adding Azure Publishing pipeline by @ianhelle in https://github.com/microsoft/msticpy/pull/838

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.16.0...v2.16.1

- Python
Published by ianhelle about 1 year ago

msticpy - Cyberint TI provider and Prisma Cloud (Palo Alto) Data provider

Prisma Cloud Driver

This pull request adds support for integrating Prisma Cloud into MSTICPy. By including a dedicated PrismaCloudDriver, the goal is to enable querying and analyzing data from Prisma Cloud’s APIs within MSTICPy’s data analysis framework.

The Prisma Cloud Driver, developed by Palo Alto Networks, integrates MSTICPy with Prisma Cloud’s security platform. It enables seamless authentication, querying, and data retrieval from Prisma Cloud’s assets, configurations, and events. By incorporating this driver, MSTICPy users gain streamlined access to cloud security data, allowing to perform in-depth threat analysis, compliance checks, and security investigations directly within their existing data analysis workflows

Big thanks to @raj-axe for this

Cyberint TI Provider

TI provider uses the Cyberint API for IoC lookup.

Azure Sentinel/Azure Monitor

We've had a bit of activity around Azure Sentinel/Azure Monitor. @JPvRiel has been digging into this and found a few bugs. They also raised the issue the current Azure monitor driver has no support for custom tables. I created an experimental driver in this release but it's not working as expected. If anyone wants to take up the sword and tackle bugs #829, #830 and #831 I would appreciate your help.

831 is specifically the problems with the experimental driver

The other two are bugs in the existing Azure Monitor/Sentinel provider. (although I'm not the support for parsing time ranges is an easy fix since we're relying on the azure.monitor.query SDK to do this conversion.

Thanks to @vx3r for this.

Certificate Authentication support for OData drivers (Defender and MSGraph)

Thanks to @FlorianBracq for this.

Other changes

Lots more typing work by our esteemed @FlorianBracq

Various fixes but some important ones: - Maxmind API change - Bokeh (should now support current Bokeh versions) - Panel (workaround for seeming bug in 1.16.1)

What's Changed

Fix typing issue for FoliumMap by @FlorianBracq in https://github.com/microsoft/msticpy/pull/814
Add Azure kusto driver typing by @FlorianBracq in https://github.com/microsoft/msticpy/pull/816
Odata certificate support by @FlorianBracq in https://github.com/microsoft/msticpy/pull/812
Fix change to maxmind API 2.6.3 by @ianhelle in https://github.com/microsoft/msticpy/pull/823
Apply typing to the Cybereason driver by @FlorianBracq in https://github.com/microsoft/msticpy/pull/813
add Cyberint TI provider by @vx3r in https://github.com/microsoft/msticpy/pull/817
Ianhelle/update to v2.16.0 by @ianhelle in https://github.com/microsoft/msticpy/pull/824
Ianhelle/az monitor search driver 2025 02 05 by @ianhelle in https://github.com/microsoft/msticpy/pull/825
Fixed autogen package by @ekzhu in https://github.com/microsoft/msticpy/pull/818
prisma_cloud driver by @raj-axe in https://github.com/microsoft/msticpy/pull/821
Updating bokeh code to support 3.4.0+ by @ianhelle in https://github.com/microsoft/msticpy/pull/826
Cyberint risk key none value by @vx3r in https://github.com/microsoft/msticpy/pull/832

New Contributors

@ekzhu made their first contribution in https://github.com/microsoft/msticpy/pull/818
@raj-axe made their first contribution in https://github.com/microsoft/msticpy/pull/821

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.15.0...v2.16.0

- Python
Published by ianhelle over 1 year ago

msticpy - Multi-dimensional plots for outliers

Highlights

Multi-dimensional plots for outliers by @Tatsuya-hasegawa

The outliers module has lived in MSTICPy for a long time but been some neglected @Tatsuya-hasegawa (hacker-T) has contributed some cool visualizations to better interpret the data. Many thanks!!!

```python import numpy as np from msticpy.analysis.outliers import identifyoutliers,plotoutlier_results

n_dimension = 7

create random numeric samples

data = np.random.rand(100, n_dimension)

calc outliers by Isolation Forest algorism

clf, Xoutliers, ypredoutliers = identifyoutliers(data, data, contamination=0.1, max_features=0.4)

featurecolumns = [f'feature{i}' for i in range(1, ndimension+1)]

plotoutlierresults( clf, data, data, Xoutliers, featurecolumns=featurecolumns, plttitle="MSTICPY Isolation Forest Anomaly Detection for Multi Dimension Features" ) ```

Improved code/docs for federated authentication for M365D/M356 Graph providers - @ryan-detect-dot-dev

Although using federated auth (rather than client secret) has been possible for a while, the documentation for how to use this was in the MSTICPy docs. Thanks to Ryan we now have this (along with cleaned up code for the Defender* data providers. (although Ryan is listed as a new contributor below - he has made several previous contributions under a different GitHub identity)

Rigorous Type Annotation work started by @FlorianBracq earlier this year continues.

This helps to make the code more robust and clearer to read and use. This is thankless work but my huge thanks go out to @FlorianBracq for this!

Other fixes

Some other important fixes to CyberReason driver and Azure Monitor/MS Sentinel driver are also included

What's Changed

Cybereason driver fix http429 tests and exception by @vx3r in https://github.com/microsoft/msticpy/pull/803
Cybereason driver query return instance name in dataframe by @vx3r in https://github.com/microsoft/msticpy/pull/804
Add multi dimension plots to analysis.outliers module. by @Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/805
Avoid httpx 0.28.0 for unit tests by @ianhelle in https://github.com/microsoft/msticpy/pull/811
Add typing hints to core classes by @FlorianBracq in https://github.com/microsoft/msticpy/pull/810
Fixing azuremonitordriver for deprecated httpx API by @ianhelle in https://github.com/microsoft/msticpy/pull/809
Update version to 2.15.0 by @ianhelle in https://github.com/microsoft/msticpy/pull/806
Update MDATP Driver for delegated auth by @ryan-detect-dot-dev in https://github.com/microsoft/msticpy/pull/784

New Contributors

@ryan-detect-dot-dev made their first contribution in https://github.com/microsoft/msticpy/pull/784

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.14.0...v2.15.0

- Python
Published by ianhelle over 1 year ago

msticpy - User Session Management, MaxMind Geolit fix, Extract nested dicts from Pandas

User Session Configuration

Do you always have one or more data providers or other components that you need to load for every notebook you create? I do, and got a bit fed up with typing the same lines of code over and over again.

User session configuration lets you specify which providers are loaded, whether or not to connect and which parameters to supply at load and connect time. You put all of this into a straightforward YAML file and load it using the following: ```python3 import msticpy as mp # you likely will already be doing this mp.init_notebook() # and this

mp.loadusersession("myconfig.yaml") # if you have a "mpusersession.yaml" in the current directory # you can skip the parameter This example shows the structure of the YAML:yaml QueryProviders: qryprovsent: DataEnvironment: MSSentinel InitArgs: debug: True Connect: True ConnectArgs: workspace: MySoc authmethods: ['cli', 'devicecode'] qryprovmd: DataEnvironment: M365D Components: mssentinel: Module: msticpy.context.azure Class: MicrosoftSentinel InitArgs: Connect: True ConnectArgs: workspace: MySoc authmethods: ['cli', 'devicecode'] ``The providers/components created (e.g.qryprov_sent` in this example) are published back to your notebook Python namespace, so you'll see these available as variables ready to use.

This configuration file is equivalent to the following code: ```python3 qryprovsent = mp.QueryProvider("MSSentinel") qryprovsent.connect(workspace="MySoc", authmethods=['cli', 'devicecode']) qryprovmd = mp.QueryProvider("M365D")

from msticpy.context.azure import MicrosoftSentinel mssentinel = MicrosoftSentinel() mssentinel.connect(workspace="MySoc", authmethods=['cli', 'devicecode']) ``Not a huge saving, on the face of it, but if you create a lot of notebooks or want to use msticpy in an automation scenario, it can be very helpful. Include averbose=Trueparameter toloadusersession` to see more detailed logging of what is going on. See the full documentation here

Maxmind GeoIPLite fix

Sometime recently (not too sure when) Maxmind changed their download procedure to use a different URL and authentication mechanism. This was causing auto-update to fail. To use the new mechanism you need to get your Maxmind User Account ID (login and look at your account properties) and add that to your msticpyconfig.yaml as shown below. yaml OtherProviders: GeoIPLite: Args: AccountID: "1234567" AuthKey: EnvironmentVar: "MAXMIND_AUTH" DBFolder: "~/.msticpy" Provider: "GeoLiteLookup"

Extract nested dictionaries from pandas column to multiple rows/columns

@pioneerHitesh has added this as a new method in the mp_pivot pandas extension: python data_df.mp_pivot.dict_to_dataframe(col="my_nested_column") It returns a dataframe with the column recursively expanded: - lists become new rows - dictionaries become new columns

So a column with the following structure:

| | NCol | |---:|:-------------------------------------------------------------| | 0 | {'A': ['A1', 'A2', 'A3'], 'B': {'B1': 'B1-1', 'B2': 'B2-1'}} | | 1 | {'A': ['A3', 'A4', 'A5'], 'B': {'B3': 'B3-1', 'B4': 'B4-1'}} |

python my_df = src_df.mp_pivot.dict_to_dataframe(col="NCol") my_df Would be unpacked to:

| | A.0 | A.1 | A.2 | B.B1 | B.B2 | B.B3 | B.B4 | |---:|:------|:------|:------|:-------|:-------|:-------|:-------| | 0 | A1 | A2 | A3 | B1-1 | B2-1 | nan | nan | | 1 | A3 | A4 | A5 | nan | nan | B3-1 | B4-1 |

What's Changed

Authentication module unit test by @ianhelle in https://github.com/microsoft/msticpy/pull/800
Use sessions config and GeoIP download failure by @ianhelle in https://github.com/microsoft/msticpy/pull/801
Added Inbuilt function to extract nested JSON by @pioneerHitesh in https://github.com/microsoft/msticpy/pull/798
Add max retry parameter to the execution prevent HTTP 429 by @vx3r in https://github.com/microsoft/msticpy/pull/802

New Contributors

@pioneerHitesh made their first contribution in https://github.com/microsoft/msticpy/pull/798
@vx3r made their first contribution in https://github.com/microsoft/msticpy/pull/802

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.13.1...v2.14.0

- Python
Published by ianhelle over 1 year ago

msticpy - Hotfix for authentication error

We introduced a bug in azureauthcore that caused Azure authentication to fail.

What's Changed

Provider and lookup typing by @FlorianBracq in https://github.com/microsoft/msticpy/pull/795
Fix for bug in azurecoreauth that fails authentication by @ianhelle in https://github.com/microsoft/msticpy/pull/799

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.13.0...v2.13.1

- Python
Published by ianhelle over 1 year ago

msticpy - AI documentation assistant, BinaryEdge TI provider and other misc fixes

We've been quietly doing some work to introduce LLM/GPT/AI capabilities into msticpy. @EileenG02 has helped us in that direction by building a document Q&A agent using Autogen.

You can try it out in a notebook using the following:

Load the magic extension ipython %load_ext msticpy.aiagents.mp_docs_rag_magic

Ask a question in a separate cell using the %%ask cell magic ipython %%ask What are the three things that I need to connect to Azure Query Provider?

Awesome work @EileenG02!

There's also a new TI provider for BinaryEdge courtesy of @petebryan.

Alongside this there have been quite a few contributions to fix and improve things like: - Splunk improvements (thanks @Tatsuya-hasegawa) - Fixes for Sentinel provider getalertrules to use updated API (thanks @BWC-TomW) - A massive amount of type annotation work and fixes to context/TI providers by @FlorianBracq - Miscellaneous fixes to things like Sentinel TI provider, MSSentinel tidy-up to more consistently handle parameters, correct use of the term CountryOrRegionName from CountryName in geolocation contexts.

The gory details of the PRs follow:

What's Changed

Add extra tests and fixes to QueryProvider, DriverBase and (as)sync query handling by @FlorianBracq in https://github.com/microsoft/msticpy/pull/777
Fix incorrect ref to ip_utils module in docs by @ianhelle in https://github.com/microsoft/msticpy/pull/779
Fix some deprecation warnings by @FlorianBracq in https://github.com/microsoft/msticpy/pull/781
Fixing np.NaN error and build warnings by @ianhelle in https://github.com/microsoft/msticpy/pull/785
Removing data matching AV signatures by @ianhelle in https://github.com/microsoft/msticpy/pull/786
Create codeql_updated.yml by @ianhelle in https://github.com/microsoft/msticpy/pull/787
Update black requirement from <24.0.0,>=20.8b1 to >=20.8b1,<25.0.0 by @dependabot in https://github.com/microsoft/msticpy/pull/742
Update docutils requirement from <0.20.0 to <0.22.0 by @dependabot in https://github.com/microsoft/msticpy/pull/768
Add upload data styles to Splunk uploader by @Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/776
Added BinaryEdge provider by @petebryan in https://github.com/microsoft/msticpy/pull/780
Update sentinelanalytics.py to update getalert_rules to use new API version by @BWC-TomW in https://github.com/microsoft/msticpy/pull/789
Fixing MSSentinel to obey parameters by @ianhelle in https://github.com/microsoft/msticpy/pull/791
Add Autogen and RAG Agent to MSTICpy by @EileenG02 in https://github.com/microsoft/msticpy/pull/793
Update TILookup and ContextLookup by @FlorianBracq in https://github.com/microsoft/msticpy/pull/794
Fix sentinel TI provider by @ianhelle in https://github.com/microsoft/msticpy/pull/797
Updating CountryName to CountryOrRegionName by @ianhelle in https://github.com/microsoft/msticpy/pull/796

New Contributors

@BWC-TomW made their first contribution in https://github.com/microsoft/msticpy/pull/789
@EileenG02 made their first contribution in https://github.com/microsoft/msticpy/pull/793

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.12.0...v2.13.0

- Python
Published by ianhelle almost 2 years ago

msticpy - Splunk and Sentinel Updates

Sentinel updates

WorkspaceConfig and Sentinel QueryProvider (azuremonitodriver) have had a few updates: - handle both old (Kqlmagic) and standard connection string formats in WorkspaceConfig - removing a lot of legacy code from WorkspaceConfig - Allow additional connection parameters to be used with MSSentinel QueryProvider for authentication parameters (e.g. you can now supply authentication parameters like "clientid", "clientsecret" to query_provider.connect()) - msticpyconfig.yaml now supports using an "MSSentinel" key in place of "AzureSentinel" - Workspace entries in msticpyconfig.yaml support an Args subkey, where you can add authentication parameters - these will be supplied to the connect() method if not overridden on the command line. Like Args sections for other providers, the values here can be text or references to environment variables or Azure Key Vault secrets. - Fix to MSSentinel API update_incident to add full properties

Splunk Updates

Added jwt authentication token expiry check.

Other fixes

Fix for vtlookup3.py - Fixed problematic way of using nestasyncio - this was causing failures when run from a langchain agent. Fix for lookup/tilookup - If the progress parameter was not passed it would still try to cancel a non-existent progress task and cause an exception. QueryProviders - Fix split query time-ranges calculation - thanks to @pjain90 for spotting this.

What's Changed

Set up CI with 1ES Azure Pipelines by @ianhelle in https://github.com/microsoft/msticpy/pull/763
Update ws_config to handle kqlmagic connection strings by @ianhelle in https://github.com/microsoft/msticpy/pull/767
Fix split query time-ranges calculation by @ianhelle in https://github.com/microsoft/msticpy/pull/762
Add support for ruff and u/p devcontainer by @ianhelle in https://github.com/microsoft/msticpy/pull/765
Add jwt auth token expire check and modify some messages when connecting Splunk by @Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/770
WSConfig updates by @ianhelle in https://github.com/microsoft/msticpy/pull/771
Pass true for props into _build_sent_data when calling update_incident by @kylelol in https://github.com/microsoft/msticpy/pull/774
Changing cert thumbprint from Sha1 to Sha256 in Az Kusto driver by @ianhelle in https://github.com/microsoft/msticpy/pull/775

New Contributors

@kylelol made their first contribution in https://github.com/microsoft/msticpy/pull/774
@pjain90 made their first contribution in https://github.com/microsoft/msticpy/pull/762

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.11.0...v2.12.0

- Python
Published by ianhelle about 2 years ago

msticpy - Sentinel Split Query fix

This is a minor release mainly to add a warning for Kusto/Sentinel queries that return partial results. A close friend of MSTICPy (thx @Cyb3r-Monk) had spotted that MSTICPy does not report partial results when doing split queries so it's possible to lose data from the query range silently.

Due to an unfortunate admin error, the fix for this was committed direct to main, so no PR for this is available. :-(

If you want the query to fail (throw an exception) rather than just warn you can supply a new parameter fail_if_partial. This only affects the Sentinel query provider and works for standard as well as split queries.

NOTE: the documentation has a typo and calls this fail_on_commit - we'll fix that in the next release to support both fail_if_partial and fail_on_partial

Example python3 qry_prov.exec_query(query_string, fail_if_partial=True)

What's Changed

Missing PR for partial query warning and fixes for pandas deprecation warnings See the diff for changes
Fixing group.apply for pandas < 2.2.1 by @ianhelle in https://github.com/microsoft/msticpy/pull/759
Added missing quotation in code block by @ryan-aus in https://github.com/microsoft/msticpy/pull/753
Bump httpx from 0.25.2 to 0.27.0 by @dependabot in https://github.com/microsoft/msticpy/pull/754
Bump readthedocs-sphinx-ext from 2.2.3 to 2.2.5 by @dependabot in https://github.com/microsoft/msticpy/pull/743
Updated conda reqs files for new packages by @ianhelle in https://github.com/microsoft/msticpy/pull/758
Build break fix for splunk SDK by @ianhelle in https://github.com/microsoft/msticpy/pull/760

New Contributors

@ryan-aus made their first contribution in https://github.com/microsoft/msticpy/pull/753

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.10.0...v2.11.0

- Python
Published by ianhelle about 2 years ago

msticpy - v2.10.0

What's Changed

Add nest_asyncio to run threaded queries by @FlorianBracq in https://github.com/microsoft/msticpy/pull/737
Bump sphinx-rtd-theme from 1.3.0 to 2.0.0 by @dependabot in https://github.com/microsoft/msticpy/pull/738
Bump httpx from 0.25.0 to 0.25.2 by @dependabot in https://github.com/microsoft/msticpy/pull/736
Adding Virus Total Search Capabilities by @secops-account in https://github.com/microsoft/msticpy/pull/739
Add security token auth and credential loading from msticpyconfig.yaml to SplunkUploader by @Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/731
fix: updated getquery_status in the azure monitor driver by @aka0 in https://github.com/microsoft/msticpy/pull/745
Added M365DGraph to the supported environments for existing queries by @d3vzer0 in https://github.com/microsoft/msticpy/pull/748
Small Typo correction in SentinelWatchlists.rst by @Korving-F in https://github.com/microsoft/msticpy/pull/746
Fix ibm_xforce TI provider for domain names and URLs by @pcoccoli in https://github.com/microsoft/msticpy/pull/749
Update python-package.yml by @ianhelle in https://github.com/microsoft/msticpy/pull/750
Ianhelle/aml updates 2024 01 31 by @ianhelle in https://github.com/microsoft/msticpy/pull/751
Ianhelle/warning fixes 2024 02 11 by @ianhelle in https://github.com/microsoft/msticpy/pull/752

New Contributors

@secops-account made their first contribution in https://github.com/microsoft/msticpy/pull/739
@aka0 made their first contribution in https://github.com/microsoft/msticpy/pull/745
@Korving-F made their first contribution in https://github.com/microsoft/msticpy/pull/746
@pcoccoli made their first contribution in https://github.com/microsoft/msticpy/pull/749

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.9.0...v2.10.0

- Python
Published by ianhelle over 2 years ago

msticpy - Defender Advanced hunting, IPQualityScore TI provider

Some of the highlights of this release:

IPQualityScore

New TI provider submitted by @petebryan - provides a lot of interesting stats on IPs.

Defender Advanced Hunting API

Thanks to @d3vzer0 our MS Defender client is now able to use the support Graph-based API rather than the legacy APIs. To use this, for the moment use the DataEnvironment name M365DGraph when you create query provider. In the next 0.x release we will switch the other aliases for M365D, MDE, MDATP to use this new interface and deprecate the existing ones.

Startup errors when running in unexpected environments.

init_notebook made some (incorrect) assumptions about when it would be running in a Synapse environment. Azure Machine Learning have recently changed their default compute to be a Synapse environment. Fixes here will correct failures due to faulty detection of environment type.

Startup fixes and perf improvements

We've optimized some of the imports done within the package at startup so msticpy should be quicker to load.

Azure env credentials fix

Although we previously supported the Azure EnvironmentCredential credential type, our implementation allowed you to use only with ClientID + ClientSecret. The changes allow it to be used with other supported credential formats - notably username + password and certificate authentication using a certificate file.

Improvements to Entities

Although these are not visible to most people, we try to keep our Entity definitions in sync with the official Microsoft "V3" entity definitions. We've added a few entity types and updated some of the attributes to bring this in line, while still allowing backwards compatible attributes to be used.

What's Changed

Ianhelle/entity updates 2023 09 01 by @ianhelle in https://github.com/microsoft/msticpy/pull/718
Ianhelle/lazy-import-init-2023-09-26 by @ianhelle in https://github.com/microsoft/msticpy/pull/717
Fix Azure env credential authentication by @ianhelle in https://github.com/microsoft/msticpy/pull/722
Update documentation for installing in isolated env by @ccianelli22 in https://github.com/microsoft/msticpy/pull/724
Bump isort to 5.12.0 in pre-commit config by @2xyo in https://github.com/microsoft/msticpy/pull/723
Remove stack trace from logging by @FlorianBracq in https://github.com/microsoft/msticpy/pull/729
fix: init_notebook and entities by @ianhelle in https://github.com/microsoft/msticpy/pull/730
Fix time span values by @FlorianBracq in https://github.com/microsoft/msticpy/pull/728
Added additional DataProvider for Advanced Hunting via Graph by @d3vzer0 in https://github.com/microsoft/msticpy/pull/725
Allow POST HTTP method by @2xyo in https://github.com/microsoft/msticpy/pull/726
Bump readthedocs-sphinx-ext from 2.2.2 to 2.2.3 by @dependabot in https://github.com/microsoft/msticpy/pull/716
Added new TI Provider - IPQualityScore by @petebryan in https://github.com/microsoft/msticpy/pull/733

New Contributors

@2xyo made their first contribution in https://github.com/microsoft/msticpy/pull/723

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.8.0...v2.9.0

- Python
Published by ianhelle over 2 years ago

msticpy - Stability release

A few bugs had crept in over the last couple of releases: some due to buggy coding, some due the world moving forward. So, many items in this release are to address these.

Among the feature improvements are the following: * Documentation and scripts from @ccianelli22 for creating a MSTICPy install for use in isolated (no Internet) environments. This is super useful for customers operating in sovereign clouds or other air-gapped high-security environments. * Added Splunk authentication method using security token rather than username/password - thanks @Tatsuya-hasegawa * Query yaml file validation by @FlorianBracq * Paging for large CyberReason queries by @FlorianBracq * Modern method to obtain cloud-specific URL endpoints for Azure services. Previously, we were relying on msrestazure, which is now deprecated for this purpose. Many thanks to @ccianelli22 for the work to do this. * Fix (by me) for a bug I'd introduced with the switch to using Azure-monitor-query library for MS Sentinel. When using a connection string with this new driver, the logic failed to parse and extract details from this correctly. Many thanks to @cindraw for reporting this bug.

What's Changed

Update mdeprocpub.pkl by @FlorianBracq in https://github.com/microsoft/msticpy/pull/709
Update Introduction.rst by @praveenjutur in https://github.com/microsoft/msticpy/pull/700
Update methodology of getting endpoints for cloud environment by @ccianelli22 in https://github.com/microsoft/msticpy/pull/704
Validation of the YAML structure of query files by @FlorianBracq in https://github.com/microsoft/msticpy/pull/660
Intsights api update by @FlorianBracq in https://github.com/microsoft/msticpy/pull/710
Fix m365d/mde hunting query options by @Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/702
Cybereason pagination support + multi-threading by @FlorianBracq in https://github.com/microsoft/msticpy/pull/707
Add bearer token auth to splunk driver by @Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/708
fix wl bug when creating a new wl when wl count is 0 by @ccianelli22 in https://github.com/microsoft/msticpy/pull/719
Update installation docs to include installation for isolated envs by @ccianelli22 in https://github.com/microsoft/msticpy/pull/715
Fixing regular expression error for connection string in WorkspaceConfig by @ianhelle in https://github.com/microsoft/msticpy/pull/706
Fix documentation formatting, update steps for downloading msticpy by @ccianelli22 in https://github.com/microsoft/msticpy/pull/720

New Contributors

@praveenjutur made their first contribution in https://github.com/microsoft/msticpy/pull/700
@ccianelli22 made their first contribution in https://github.com/microsoft/msticpy/pull/704

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.7.0...v2.8.0

- Python
Published by ianhelle over 2 years ago

msticpy - 2.8.0 pre-release

Updated method to dynamically fetch Azure endpoints (rather than relying on deprecated msrestazure). Updated version of Insight data provider

- Python
Published by ianhelle over 2 years ago

msticpy - TI Providers, Sentinel/Kusto Drivers, Query Editor

Main Changes in this release

Two new TI Providers

Two cool new providers to add to the growing family in MSTICPy:

CrowdSec is a commercial Malicious IP threat service with a free tier for limited threat lookups. (big thanks to @sbs2001 for submitting this)
AbuseIPDB - is an open/free provider of threat intel on malicious IP addresses, providing a central abuse list to lookup IP addresses that have been associated with malicious activity. (big thanks to @rrevuelta for submitting this.)

As with other providers, these are automatically enabled for use if you include settings for the API keys in your msticpyconfig.yaml

Updated Data providers for Sentinel/Azure Monitor/Log Analytics and Kusto/Azure Data Explorer

In v2.5.0 we introduced replacement drivers for the MS Sentinel/LogAnalytics/Azure Monitor and Kusto/Azure Data Explorer providers.

The new drivers are based on the Azure SDKs for each data service. You can read the release notes for them here.

The new drivers give several advantages, like being able to run queries across multiple workspaces or Kusto clusters in parallel. Splitting large queries by time chunks (split_query_by parameter) will also run multiple segments in parallel, dramatically speeding up the query. The default parallelism is 4 simultaneous threads but you can change this (although be wary of the impact on the data service for highly parallel queries - this may affect other users and services accessing the data).

The new drivers are now the default drivers for these providers. They are used by default for the "MSSentinel" and "Kusto" data environment identifiers. For backward compatibility, they will also continue to support the "MSSentinelNew" and "KustoNew" identifiers.

To invoke the previous Kqlmagic-based drivers use "MSSentinelLegacy" or "KustoLegacy".

This change also brings a dependency change for MSTICPy. The following packages are now part of the core installed dependencies: - azure-kusto-data - azure-monitor-query

Kqlmagic and its dependencies are no longer installed by default but can be installed with the "kql" extra: bash python -m pip install msticpy[kql]

See these links to read more about the MSSentinel provider and Kusto providers.

Query Editor

We've added an ipywidgets based query template editor .

note: this is somewhat provisional so please be sure to test and report bugs.

The query editor allows you to edit existing query files or create new ones and helps manage the various query properties (like parameter definitions) and query metadata.

Check out the documentation on how to use this in the Extending section of the MSTICPy documentation.

Updates to Authentication.

The improvements here mainly affect the AzureData and MicrosoftSentinel classes but' also bring some improvements to the core authentication - such as being able to specify the Azure cloud from the az_connect function and authenticate by providing an AzureCredential.

You can now authenticate by supplying an AzureCredential as a credential parameter for AzureData and MicrosoftSentinel connect methods.
The connect methods for both these classes also support cloud parameter to specify different sovereign clouds
The __init__ and connect methods are instrumented with logging to help debug issues:

```python3 import msticpy as mp from msticpy.context.azure.sentinel_core import MicrosoftSentinel

mp.setlogginglevel("INFO") mssentinel = MicrosoftSentinel() mssentinel.connect() ```

Other major items

MS Sentinel delete watchlist API added by @mbabinski
Splunk fixes added by @Tatsuya-hasegawa

Thanks

Our thanks to the following folks who contributed to this release. @FlorianBracq @sbs2001 @rrevuelta @mbabinski @Tatsuya-hasegawa

What's Changed

Add CrowdSec TIProvider by @sbs2001 in https://github.com/microsoft/msticpy/pull/673
Added deletewatchlistitem method by @mbabinski in https://github.com/microsoft/msticpy/pull/682
Update pandas requirement from <2.0.0,>=1.4.0 to >=1.4.0,<3.0.0 by @dependabot in https://github.com/microsoft/msticpy/pull/653
Bump sphinx from 6.1.3 to 7.1.0 by @dependabot in https://github.com/microsoft/msticpy/pull/686
Add AbuseIPDB TIProvider by @rrevuelta in https://github.com/microsoft/msticpy/pull/687
Typo corrections in queries by @ianhelle in https://github.com/microsoft/msticpy/pull/684
Ianhelle/query editor 2023 04 21 by @ianhelle in https://github.com/microsoft/msticpy/pull/685
Few fix splunk driver by @Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/688
Ianhelle/mssentinel auth 2023 08 01 by @ianhelle in https://github.com/microsoft/msticpy/pull/690
Updating timeline docs to prioritize pd accessors by @ianhelle in https://github.com/microsoft/msticpy/pull/691
Fix splunk uploader create index option by @Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/692
v2.7.0 - changing new kql/sentinel drivers to be defaults by @ianhelle in https://github.com/microsoft/msticpy/pull/696

New Contributors

@sbs2001 made their first contribution in https://github.com/microsoft/msticpy/pull/673
@mbabinski made their first contribution in https://github.com/microsoft/msticpy/pull/682

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.6.0...v2.7.0

- Python
Published by ianhelle almost 3 years ago

msticpy - DataProviders, QueryEditor, CrowdSec and AbuseIPDB TIProviders

Preview release of 2.7.0

More detailed release notes in the full release.

Main Changes

Two new TI Providers:

CrowdSec (thanks to @sbs2001)
AbuseIPDB (thanks to @rrevuelta)

Updated Data providers for Sentinel/Azure Monitor/Log Analytics and Kusto/Azure Data Explorer

These were introduced in v2.5.0 but are now the default drivers for these providers.

Query Editor

ipywidgets based query template editor - this is somewhat provisional so please be sure to test and report bugs.

Updates to Authentication - esp for the AzureData and MicrosoftSentinel API modules

You can now authenticate by supplying an AzureCredential as a credential parameter
The connect methods for these support cloud parameter to specify different sovreign clouds
The init and connect methods are instrumented with logging to help debug issues:

```python3 import msticpy as mp from msticpy.context.azure.sentinel_core import MicrosoftSentinel

mp.setlogginglevel("INFO") mssentinel = MicrosoftSentinel() mssentinel.connect() ```

Other items

MS Sentinel delete watchlist API added by @mbabinski
Splunk fixes added by @Tatsuya-hasegawa

What's Changed

Add CrowdSec TIProvider by @sbs2001 in https://github.com/microsoft/msticpy/pull/673
Added deletewatchlistitem method by @mbabinski in https://github.com/microsoft/msticpy/pull/682
Update pandas requirement from <2.0.0,>=1.4.0 to >=1.4.0,<3.0.0 by @dependabot in https://github.com/microsoft/msticpy/pull/653
Bump sphinx from 6.1.3 to 7.1.0 by @dependabot in https://github.com/microsoft/msticpy/pull/686
Add AbuseIPDB TIProvider by @rrevuelta in https://github.com/microsoft/msticpy/pull/687
Typo corrections in queries by @ianhelle in https://github.com/microsoft/msticpy/pull/684
Ianhelle/query editor 2023 04 21 by @ianhelle in https://github.com/microsoft/msticpy/pull/685
Few fix splunk driver by @Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/688
Ianhelle/mssentinel auth 2023 08 01 by @ianhelle in https://github.com/microsoft/msticpy/pull/690
Updating timeline docs to prioritize pd accessors by @ianhelle in https://github.com/microsoft/msticpy/pull/691
Fix splunk uploader create index option by @Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/692
v2.7.0 - changing new kql/sentinel drivers to be defaults by @ianhelle in https://github.com/microsoft/msticpy/pull/696

New Contributors

@sbs2001 made their first contribution in https://github.com/microsoft/msticpy/pull/673
@mbabinski made their first contribution in https://github.com/microsoft/msticpy/pull/682

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.6.0...v2.7.0.pre1

- Python
Published by ianhelle almost 3 years ago

msticpy - v2.6.0 Parallel Queries, Velociraptor data

The three big changes in this release are: * Executing MS Sentinel and Kusto queries in parallel across multiple instance * Threaded (parallel) execution of time-split queries * Addition of data provider to query local (exported) Velociraptor logs

Many thanks to @d3vzer0 for inspiration and early work on the threaded query feature. Many thanks @juju4 for inspiration and work on the Velociraptor support.

Support for running a query across multiple connections (with optional threaded operation)

It is common for data services to be spread across multiple tenants or workloads. E.g., multiple Sentinel workspaces, Microsoft Defender subscriptions or Splunk instances. You can use the MSTICPy QueryProvider to run a query across multiple connections and return the results in a single DataFrame.

To create a multi-instance provider: * Create an instance of a QueryProvider for your data source and execute the connect() method to connect to the first instance of your data service. * Then use the add_connection() method. This takes the same parameters as the connect() method (the parameters for this method vary by data provider) to add additional instance connections.

add_connection() also supports an alias parameter to allow you to refer to the connection by a friendly name.

```python3

qry_prov = QueryProvider("MSSentinel")
qry_prov.connect(workspace="Workspace1")
qry_prov.add_connection(workspace="Workspace2, alias="Workspace2")
qry_prov.list_connections()

``` When you now run a query for this provider, the query will be run on all of the connections and the results will be returned as a single dataframe.

```python3

test_query = '''
    SecurityAlert
    | take 5
    '''

query_test = qry_prov.exec_query(query=test_query)
query_test.head()

```

Some of the MSTICPy drivers support asynchronous execution of queries against multiple instances, so that the time taken to run the query is much reduced compared to running the queries sequentially. Drivers that support asynchronous queries will use this automatically. The initial set of multi-threaded drivers are: - MSSentinelNew (the new version of the MSSentinel driver) - KustoNew (the new version of the Kusto/Azure Data Explorer driver)

By default, the queries will use at most 4 concurrent threads. You can override this by initializing the QueryProvider with the max_threads parameter to set it to the number of threads you want. Although you should be cautious about using too many simultaneous connections due to the potential impact on the cluster performance.

```python3

qry_prov = QueryProvider("MSSentinel", max_threads=10)

```

Multi-threaded support for split/shared queries

MSTICPy has supported splitting large queries by time-slice for a while. This is documented here Splitting a Query into time chunks. With this release, we've added asynchronous support for this (if the driver supports threaded/async operation) so that multiple chunks of the query will run in parallel.

```python3

qry_prov.SecurityAlert.list_alerts(start=start, end=end, split_by="1d")

``Use the parametersplitquerybyorsplit_by` to specify a time range (the time unit uses the same syntax as pandas time intervals - e.g. "1D", "4h", etc. - the the pandas documentation for more details on this).

In this release sharding is also supported for ad hoc queries as long as you add "start" and "end" parameters to the query (this is still experimental, so let us know if you have issues with this).

Velociraptor Local Data Provider

The Velociraptor data provider can read Velociraptor log files and provide convenient query functions for each data set in the output logs.

The provider can read files from one or more hosts, stored in in separate folders. The files are read, converted to pandas DataFrames and grouped by table/event. Multiple log files of the same type (when reading in data from multiple hosts) are concatenated into a single DataFrame.

To use the Velociraptor provider, you need to create an QueryProvider instance, passing the string "Velociraptor" (or "VelociraptorLogs") as the data_environment parameter. You also need to add the data_paths parameter to specify specific folders that you want to search for log file (although you can set these paths in msticpyconfig.yaml, if you do this frequently).

You can specify multiple folders to have the logs from different hosts.

python3 qry_prov = mp.QueryProvider("VelociraptorLogs", data_paths=["~/my_logs"]) Calling the connect method triggers the provider to read the locations of the log files (although the contents are not read until a query function is run).

```python3

qry_prov.connect()

Listing Velociraptor tables

python3 qry_prov.list_queries() ['velociraptor.Custom_Windows_NetBIOS', 'velociraptor.Custom_Windows_Patches', 'velociraptor.Custom_Windows_Sysinternals_PSInfo', 'velociraptor.Custom_Windows_Sysinternals_PSLoggedOn', ....

Each query returns the table of data types retrieved from the logs.

```python3

qry_prov.vc_prov.velociraptor.Windows_Forensics_ProcessInfo()

```

| Name | PebBaseAddress | Pid | ImagePathName | CommandLine | CurrentDirectory | Env |
| :------ | :--------------- | ----: | :----------- | :---------------- | :----------------- | :---- | | LogonUI.exe | 0x95bd3d2000 | 804 | C:\Windows\system32\LogonUI.exe | "LogonUI.exe" /flags:0x2 /state0:0xa3b92855 /state1:0x41c64e6d | C:\Windows\system32\ | {'ALLUSERSP | | dwm.exe | 0x6cf4351000 | 848 | C:\Windows\system32\dwm.exe | "dwm.exe" | C:\Windows\system32\ | {'ALLUSERSP | | svchost.exe | 0x6cd64d000 | 872 | C:\Windows\System32\svchost.exe | C:\Windows\System32\svchost.exe -k termsvcs | C:\Windows\system32\ | {'ALLUSERSP | | svchost.exe | 0x7d18e99000 | 912 | C:\Windows\System32\svchost.exe | C:\Windows\System32\svchost.exe -k LocalServiceNetworkRestricted | C:\Windows\system32\ | {'ALLUSERSP | | svchost.exe | 0x5c762eb000 | 920 | C:\Windows\system32\svchost.exe | C:\Windows\system32\svchost.exe -k LocalService | C:\Windows\system32\ | {'ALLUSERSP |

What's Changed

Ianhelle/velociraptor provider 2023 05 19 by @ianhelle in https://github.com/microsoft/msticpy/pull/668
Updating github checkout and upload-artifact to v3 by @ianhelle in https://github.com/microsoft/msticpy/pull/669
Added multithreading support for additional connections (+fixes) by @d3vzer0 in https://github.com/microsoft/msticpy/pull/645
Bump readthedocs-sphinx-ext from 2.2.0 to 2.2.2 by @dependabot in https://github.com/microsoft/msticpy/pull/679
Bump sphinx-rtd-theme from 1.2.0 to 1.2.2 by @dependabot in https://github.com/microsoft/msticpy/pull/675
Bump httpx from 0.24.0 to 0.24.1 by @dependabot in https://github.com/microsoft/msticpy/pull/666
Ianhelle/fix func query names 2023 06 30 by @ianhelle in https://github.com/microsoft/msticpy/pull/680

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.5.3...v2.6.0

- Python
Published by ianhelle almost 3 years ago

msticpy - v2.5.3: ipwidgets, Sentinel and Kusto driver fixes

Minor release addressing the following: * Azure-monitor-query release 1.2.0 changed the format of the endpoint URLs that it accepts. Fixed the azurekusto driver (currently invoked with the "KustoNew" data environment) so that it will provide the correct format for 1.2.0+ and pre 1.2.0 versions * Bug in the kqldriver (MS Sentinel) was causing the kustodriver to fail when querying. The latter is a subclass of the former and was failing due to an attribute that was defined in the parent (kqldriver) but not in the child (kustodriver). This affected the older (current) Kusto driver version and does not affect the new azurekusto ("KustoNew") driver. * Updated requirements to allow ipywidgets 8.x to install by default (this is now supported by vs code * Updated documentation for the new Sentinel and Kusto drivers to add instructions for manually installing the required SDK components (azure-monitor-query and azure-kusto-data)

What's Changed

Azure monitor endpoint URL has changed format in v1.2.0 by @ianhelle in https://github.com/microsoft/msticpy/pull/677

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.5.2...v2.5.3

- Python
Published by ianhelle almost 3 years ago

msticpy - v2.5.2: Hotfix for Holoviz panel compatibility

Release is mainly to align bokeh version requirements with the new release of Holoviz panel.
- moved bokeh from <3.0.0 to < 4.0.0
Also fixes an issue with the MicrosoftSentinel attribute disappearing from msticpy

What's Changed

Ianhelle/hotfix 2.5.2 2023 06 08 by @ianhelle in https://github.com/microsoft/msticpy/pull/676

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.5.1...v2.5.2

- Python
Published by ianhelle about 3 years ago

msticpy - v2.5.1: Hotfix for import failure

Some minor fixes that address: - importing msticpy without some non-default azure packages installed failed - added more resiliency to query reader so that the whole thing does not fail if there is bad query file. - removed initialization dependency on azure-resourcegraph in MicrosoftSentinel class.

What's Changed

Hotfix for v2.5.1 by @ianhelle in https://github.com/microsoft/msticpy/pull/672

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.5.0...v2.5.1

- Python
Published by ianhelle about 3 years ago

msticpy - v2.5.0

Summary of main changes

New MS Sentinel and Azure Kusto drivers/data providers - these include support for multi-threaded parallel queries, proxies and user-defined query timeouts.
Extensibility model for MSTICPy - you can create private data providers, TI and Context providers and load them into MSTICPy alongside the built-in providers.
MS Sentinel repo query download - add current detection and hunting queries from the Sentinel repo as Sentinel queries runnable from MSTICPy/notebooks
OSQuery data provider - makes it easy to import OS Query logs to dataframes to do additional processing/analysis on them.
Panel tabulator now supported as default data viewer (a million times better than the one we built!)

More details on these changes below

Sentinel and Kusto provider new drivers

This change adds replacement drivers for the MSSentinel and Kusto data providers. In place of Kqlmagic, these drivers use the azure-kusto-data and azure-monitor-query SDKs, respectively.

Currently these drivers are enabled alongside the existing versions - in a future version we will make these the defaults for Sentinel and Kusto.

Some of the main changes with these new versions:

They use the provider names MSSentinel_New and Kusto_New when creating a QueryProvider instance.
Both drivers support setting proxies for firewall-protected networks
Both drivers support custom configuration of the server timeout via a timeout parameter
Both drivers use integrated Azure authentication by default and support the auth_types and tenant_id parameters used elsewhere in MSTICPy
Both drivers support threaded execution for parallelizing queries (across multiple workspaces/clusters or split by time) - this functionality, however, will be exposed in v2.6.0 via a separate feature.
The MSSentinel_New driver allows you to execute the same query across multiple workspaces in parallel and returns the results as a combined dataframe.
Some of the previous parameters have been deprecated:
- mp_az_auth is replaced by auth_types (the former still works but will be removed in a future release).
- mp_az_auth_tenant_id is replaced by tenant_id (the former is not supported in the new providers).

Note: in order to use these new versions you must have the azure-kusto-data and/or azure-monitor-query Python packages installed. You can install these using pip install msticpy[azure] or install them separately using pip.

For more details on how to use these providers, see: * Documentation for the new Sentinel provider * Documentation for the new Kusto provider

Changes specific to the MS Sentinel provider

Connecting to multiple workspaces allows you to run queries across these workspaces and return the combined results as a single Pandas DataFrame. The workspaces must use common authentication credentials and should have the same data schema. ```python3

use workspace names if these workspaces are configured in msticpyconfig.yaml

qry_prov.connect(workspaces=["Default", "MyOtherWorkspace"])

or use a list of workspace IDs

qry_prov.connect(workspaces=["e6b4bc15-119b-45a2-8f3d-c39ed384ed37", "b17e0e5a...."])

run query against connected workspaces

qryprov.SecurityAlert.listalerts() ```

Changes specific to the Kusto provider

The settings format has changed (although the existing format is still supported albeit with some limited functionality). See the Kusto provider documentation for details.
In the earlier implementation of driver you can specify a new cluster to connect to in when executing a query. This is no longer supported. Once the provider is connected to a cluster it will only execute queries against that cluster. (You can however, call the connect() function to connect the provider to a new cluster before running the query.)
Filtering pre-defined queries by cluster. If you have MSTICPy query definitions for the Kusto provider, these will all be attached as methods of the QueryProvider, when it is created. However, as soon as you connect to a specific cluster, the queries will be filtered down to show only the queries that are intended to run on that cluster.
New APIs (exposed via the query_provider):
- getdatabasenames() - return list of databases for the connected cluster
- getdatabaseschema() - return table schema for a database in the cluster
- configured_clusters() - return a list of clusters configured in msticpyconfig.yaml
- set_cluster() - switch connected to cluster to a different one (you can use the connect method to do this, which also lets you specify additional connection parameters).

Extend MSTICPy with Data provider, TI provider and Context provider plugins

This adds the ability to "side-load" data providers, TI providers and context providers. If you have a data/TI/context source that you want to use in MSTICPy you can write a provider (deriving from one of the base provider classes) and tell MSTICPy where to load it from.

In a future release we'll build on this framework to let you install plugins from external packages and provide some cookie-cutter templates to generate skelton provider classes.

Writing a TI provider or Context provider (partial example)

```python

class TIProviderHttpTest(HttpTIProvider):
    """Custom IT provider TI HTTP."""

    PROVIDER_NAME = "MyTIProvider"
    _BASE_URL = "https://api.service.com"
    _QUERIES = _QUERIES = {
        "ipv4": APILookupParams(path="/api/v1/indicators/IPv4/{observable}/general"),
        "ipv6": APILookupParams(path="/api/v1/indicators/IPv6/{observable}/general"),

```

Telling MSTICPy to load the plugins

Load on demand ```python

import msticpy as mp

mp.load_plugins(plugin_paths="/my_modules")

# or multiple paths
mp.load_plugins(
    plugin_paths=["./my_modules", "./my_other_modules"]
)

```

Or specify in msticpyconfig.yaml

yaml ... Custom: - "testdata" PluginFolders: - tests/testdata/plugins Azure: ... See the new Extending Msticpy section in our docs. If you want to contribute any of the drivers you write, also check out the new Development section in the MSTICPy docs.

OS Query Provider

Great contribution from @juju4 here (with a bit of collaboration with @ianhelle). Create a MSTICPy QueryProvider with the data environment name "OSQueryLogs" and load forensic logs from OSQuery.

```python3

specify one or more paths to folders where the dumped JSON OSQuery logs can be found

qryprov = mp.QueryProvider("OSQueryLogs", datapaths=["~/logs1", "~/logs2"]) qryprov.connect() qryprov.listqueries() ['osquery.acpitables', 'osquery.devicenodes', 'osquery.dnsresolvers', 'osquery.events', 'osquery.fim', 'osquery.last', 'osquery.listeningports', 'osquery.loggedinusers', 'osquery.mounts', 'osquery.opensockets', ... Each event type is available as a separate function that returns a pandas DataFrame with the combined events from the logs for that typepython3 qry_prov.osquery.processes() ```

Downloading Sentinel Detection and Hunting queries for the Sentinel Query Provider

We haven't finished documenting this or integrating it fully, so will leave the full announcement of this until the next release. If you want to play around with it look at the following module: ```python3 from msticpy.data.drivers.sentinelqueryreader import downloadandwritesentinelqueries

downloadandwritesentinelqueries( querytype="Hunting", # or "Detections" yamloutputfolder="./sentinelhunting", ) qryprov = mp.QueryProvider("SentinelNew", querypaths=["./sentinelhunting"]) ``` Since there are lots of queries, the import might take a little while in its current form.

Panel Tabulator now available as a DataViewer control.

HoloViz Panel is a powerful Bokeh-based data exploration & web app framework for Python. It has an immense amount of functionality that you can read about at the Panel documentation site. You need to have panel installed for the Tabulator-based viewer to run (pip install panel).

Unfortunately, the documentation for our Tabulator view never made it into this release but most of the functionality should be obvious from the UI. There are some useful load-time parameters that you can use at startup for things like: * selecting an initial column set. * adding columns to a per-row expando pane - useful for viewing long column values such as command-line.

We also kept the column chooser widget from the previous data viewer so that you can interactively select which columns to display. The Tabulator MSTICPy initialization parameters are documented in the code.

Most of the Tabulator init parameters are also passed through to the underlying control - which give you an immense amount of control over the viewer. These are described in the Panel Tabulator documentation

Big thanks to our contributors in this release!

@juju4 @jannieli @ianhelle @Tatsuya-hasegawa @FlorianBracq @danielyates2 @petebryan @ashwin-patil

What's Changed PR Reference

Updated Sentinel incident docs to reflect filtering options by @petebryan in https://github.com/microsoft/msticpy/pull/648
Read the docs update for Managed spark installation by @ashwin-patil in https://github.com/microsoft/msticpy/pull/647
Added documentation for the polling detection module by @danielyates2 in https://github.com/microsoft/msticpy/pull/601
Add PyVis panel version of DataViewer. by @ianhelle in https://github.com/microsoft/msticpy/pull/646
add LocalOsquery driver based on LocalData one by @juju4 in https://github.com/microsoft/msticpy/pull/624
Bump httpx from 0.23.3 to 0.24.0 by @dependabot in https://github.com/microsoft/msticpy/pull/655
Sentinel and Kusto new providers by @ianhelle in https://github.com/microsoft/msticpy/pull/656
Fix a critical bug of Splunk results reader, lack of pagination by @Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/657
Update azurekustodriver.py by @FlorianBracq in https://github.com/microsoft/msticpy/pull/664
Ianhelle/mp extensibility 2023 02 09 by @ianhelle in https://github.com/microsoft/msticpy/pull/632
Format of cluster name has changed in new KustoClient. by @ianhelle in https://github.com/microsoft/msticpy/pull/667
Write Sentinel queries to YAML for Github Browser by @jannieli in https://github.com/microsoft/msticpy/pull/491

New Contributors

@danielyates2 made their first contribution in https://github.com/microsoft/msticpy/pull/601
@Tatsuya-hasegawa made their first contribution in https://github.com/microsoft/msticpy/pull/657
@jannieli made their first contribution in https://github.com/microsoft/msticpy/pull/491

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.4.0...v2.5.0

- Python
Published by ianhelle about 3 years ago

msticpy - v2.4.0

Main changes for this release

There are no huge changes in this release but a good variety of important updates and fixes. We're also delighted to welcome 3 new contributors to the MSTICPy family: * @ZeArioch * @ctoma73 * @jllangley

Thanks so much!

New Threat Intel provider for Pulsedive from @fr0gger #609

This includes a standard MSTICPy TI provider (so you can include it in you collection of providers used for regular TI checks on IPs, URLs, etc. This provider also contain a few custom methods that let to query some other facets of the Pulsedive data. For example, the explore function that allows you to use the pulsedive query language python pddetail = pdlookup.explore(query="ioc=pulsedive.com or threat=AgentTesla") pddetail You can also request a can on a domain or URL python pdscan = pdlookup.scan(observable= "alvoportas.com.br") pdscan To use any of the Pulsedive features you'll need an account and API key from Pulsedive See more details of the usage in the Pulsedive notebook

Process tree updates #637

@ZeArioch added Process Tree support for FireEye HX data so it should be automatically recognized and render correct
We also added the ability to export a process tree as a text object - which is useful if you want to copy and paste a tree or part of it into a non-HTML document. See the Process Tree docs for more details +-- Process: C:Program FilesMicrosoft Monitoring AgentAgentMonitoringHost.exe PID: 0x888 Time: 1970-01-01 00:00:00+00:00 Cmdline: nan Account: nan LoginID: 0x3e7 +-- Process: C:WindowsSystem32cscript.exe PID: 0x364 Time: 2019-01-15 04:15:26+00:00 Cmdline: "C:Windowssystem32cscript.exe" /nologo "MonitorKnowledgeDiscovery.vbs" Account: WORKGROUPMSTICAlertsWin1$ LoginID: 0x3e7 +-- Process: C:Program FilesMicrosoft Monitoring AgentAgentHealth Service StateCT_602681692NativeDSCDesiredStateConfigurationASMHost.exe PID: 0x1c4 Time: 2019-01-15 04:16:24.007000+00:00 Cmdline: "C:Program FilesMicrosoft Monitoring AgentAgentHealth Service StateCT_602681692NativeDSCDesiredStateConfigurationASMHost.exe" GetInventory "C:Program FilesMicrosoft Monitoring AgentAgentHealth Service StateCT_602681692workServiceStateServiceState.mof" "C:Program FilesMicrosoft Monitoring AgentAgentHealth Service StateCT_602681692workServiceState" Account: WORKGROUPMSTICAlertsWin1$ LoginID: 0x3e7

Miscellaneous fixes #644

This sounds like a small item but contain several important fixes: - Azure authentication (azconnect) now avoids throwing exceptions if you ask it to use authentication types (e.g. clientsecret) where parameters are not passed (or available in environment variables). It will now just ignore those credential types and only throw an exception if no usable credential types remain. - Updates to API documentation - A new IPython magic "%saveto_cell" - this lets you save a Python object (e.g. a DataFrame to a base64-encoded blob in a new cell. The cell contains code to restore the original data. This is subject to the usual caveats about pickle - including the security ones. Do Not run a cell that unpickles some arbitrary data in notebooks that you do not trust. - A bunch of changes/fixes to the Sentinel APIs - Most of these are fixes related to the newly-supported Sentinel Dynamic Summaries feature - Some minor fixes also to Sentinel core

Python Logging support #640

We should have had this from the beginning but it's never too late to start correcting your mistakes. We've implemented a central logging module and started to instrument some of the code that is especially complex and where people often get stuck with cryptic errors. E.g. the init_notebook function. We also enabled in in the authentication modules (az_connect) in #644 Most of the time, this will be invisible. However, if you need it you can just do the following: ```python

import msticpy as mp # if not already imported

mp.setlogginglevel("INFO") ``Then re-run the function that you are having trouble with again. You can also use theMSTICPYLOGLEVELvariable to control this. And, if you want to log to a file, set the env variableMSTICPYLOGFILE` to the path of your log file. (You'll need to restart the kernel/python session and reload MSTICPy for this to take effect).

Support for Bokeh 3.0 #630 #642 and #650

@ctoma73 did some awesome work to track down problems with compatibility with Bokeh 3.0 and fix all of them (a lot were tedious mypy/linting fixes due to some of the more dynamic nature of the Bokeh 3.0 object model). You'll notice in #650 that we still have Bokeh 2.4.3 in the MSTICPy requirements. We're not going to change that just yet since we want compatibility with PyViz/HoloViz panel - you will likely see some panel-related features in the next minor release. Despite this (and assuming you can ignore some pip warning about MSTICPy not being compatible with Bokeh 3.x) you can install Bokeh 3.0 after MSTICPy and enjoy the delights of the new release. All of our code should be compatible (tested with 3.0.0 and 3.1.0).

That's all for this release. We'll likely be doing a follow-on 2.5.0 release that will include several contributions from our 2023 Hackmonth (which turned into a HackNMonths event).

What's Changed

Add support for FireEye HX acquisition packages in process_tree by @ZeArioch in https://github.com/microsoft/msticpy/pull/616
Adding Pulsedive as Threat Intel provider by @fr0gger in https://github.com/microsoft/msticpy/pull/609
Fix error when latest version 3.0.3 of bokeh is installed by @ctoma73 in https://github.com/microsoft/msticpy/pull/630
Adding logging and updating settings access by @ianhelle in https://github.com/microsoft/msticpy/pull/640
ProcTree and init_notebook fixes by @ianhelle in https://github.com/microsoft/msticpy/pull/637
Adding data query paths test for DEX support by @ianhelle in https://github.com/microsoft/msticpy/pull/638
Fixing RangeTool with bokeh 3.1.0 not a GestureTool by @ctoma73 in https://github.com/microsoft/msticpy/pull/642
Modified the upload_df method to split the data into batches of 10,00… by @jllangley in https://github.com/microsoft/msticpy/pull/633
Misc updates for 2.3.2 release: by @ianhelle in https://github.com/microsoft/msticpy/pull/644
Reverting to bokeh version 2.4.3 for default install by @ianhelle in https://github.com/microsoft/msticpy/pull/650

New Contributors

@ZeArioch made their first contribution in https://github.com/microsoft/msticpy/pull/616
@ctoma73 made their first contribution in https://github.com/microsoft/msticpy/pull/630
@jllangley made their first contribution in https://github.com/microsoft/msticpy/pull/633

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.3.1...v2.4.0

- Python
Published by ianhelle about 3 years ago

msticpy - MSTICPy Feb 2023 Fixes

This is minor release with mostly fixes.

Some higlights from the #631 PR

629 - You can now suppress progress bar for Threat Intel lookups (useful to avoid screen mess

when running multiple lookups from other code) python tilookup.lookup_iocs(data, progress=False)

572 - We've had a long-running issue in Azure Machine Learning where the UI does not correctly

handle javascript written by the notebook. This results in JS code in the output cells. While we're waiting for AML to re-adopt the latest Azure Notebooks package and get rid of this bug altogether we've added a fix to suppress javascript text for out Kqlmagic data provider

Fix to Azure ML use - automatic creation of msticpyconfig.yaml was writing the file to the wrong place, so users always got the message that no config file was found.
We had a request (again for batch jobs) to remove automatic display of license information in the geoip module.
Using MSTICPy offline or in isolated environment - it has always been our goal to support this but we recently discovered that we were running a check_version call from init_notebook. This function did not handle network failure and crashed the entire init_notebook process. This has been fixed so should be runnable offline or in air-gapped networks.
Related to this we've also cleaned up remaining units tests that make outbound network requests.

Full Changelist

Adding job to file issue if main build fails. by @ianhelle in https://github.com/microsoft/msticpy/pull/613
Removing prospector from CI build by @ianhelle in https://github.com/microsoft/msticpy/pull/619
Reverting PR #496 - Removing blank sub-id from resource graph list by @ianhelle in https://github.com/microsoft/msticpy/pull/621
Resolved issues with nextLink following in Sentinel API calls by @petebryan in https://github.com/microsoft/msticpy/pull/617
Fix MDE procschema by @rrevuelta in https://github.com/microsoft/msticpy/pull/626
Bump sphinx-rtd-theme from 1.1.1 to 1.2.0 by @dependabot in https://github.com/microsoft/msticpy/pull/628
Bump sphinx from 5.3.0 to 6.1.3 by @dependabot in https://github.com/microsoft/msticpy/pull/610
Ianhelle/misc fixes 2023 02 17 by @ianhelle in https://github.com/microsoft/msticpy/pull/631

New Contributors

@rrevuelta made their first contribution in https://github.com/microsoft/msticpy/pull/626

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.2.3...v2.3.1

- Python
Published by ianhelle over 3 years ago

msticpy - v2.3.0

Some new data-related features in this release. - Support for the new (still in preview at time of writing) Dynamic Summaries feature of MS Sentinel - Added ability to create and use "ad-hoc" parameterized queries for data providers - Simple search mechanism for finding queries - Support for JSON queries for CyberReason

Support for Microsoft Sentinel Dynamic Summaries

Dynamic Summaries are a Sentinel feature that allow you to persist results of query jobs in a summarized/serialized form. This might be useful for keeping results of daily watch jobs, for example. We will be using it in MSTICPy notebooks to publish more complex result sets from automated notebook runs.

MSTICPy operations available include:

Retrieve list of current dynamic Summaries
Retrieve a full dynamic summary
Create a dynamic summary
Delete a dynamic summary
Update an existing dynamic summary

Examples: ```python

list dynamic summaries

sentinel.listdynamicsummaries()

create a dynamic summary in Sentinel

sentinel.connect() sentinel.createdynamicsummary( name="MyXYZSummary", description="Summarizing the running of the XYZ job.", data=summarydf, tactics=["discovery", "exploitation"], techniques=["T1064", "T1286"], searchkey="host.domain.dom", ) ```

The MSTICPy support also includes a DynamicSummary class that lets you manipulate dynamic summary objects more easily ```python # can also import the class directly # from msticpy.context.azure.sentineldynamic import DynamicSummary # dynsummary = DynamicSummary(....) # This example shows using the "factory" method - newdynamicsummary dynsummary = sentinel.newdynamicsummary( summaryname="My new summary", summarydescription="Description of summary", sourceinfo={"TI Records": "misc"}, summaryitems=tisummarydf, ) # Add the local summary object to add to the Sentinel dynamic summaries. sentinel.createdynamicsummary(dynsummary)

Retrieve a dynamic summary from Sentinel

dynsummary = sentinel.getdynamicsummary( summaryid="cea27320-829c-4654-bbf0-b14367483418" )

the return value is a DynamicSummary object

dynsummary DynamicSummary(id=cea27320-829c-4654-bbf0-b14367483418, name=test2, items=0) ``By defaultgetdynamic_summary` returns the header data for the summary.

The next example shows how you can also fetch full data for the dynamic summary (by adding summary_items=True). From the returned object, you can convert the summary items to a pandas DataFrame.

Note: fetching summary items is done via the Sentinel QueryProvider since the APIs do not support retrieving these.

```python dynsummary = sentinel.getdynamicsummary( summaryid="cea27320-829c-4654-bbf0-b14367483418", summary_items=True )

dynsummary.todf() ```

index | Ioc | IocType | QuerySubtype | Provider | Result | Severity | Details | TimeGenerated -- | -- | -- | -- | -- | -- | -- | -- | -- OTX | hXXp://38[.]75[.]37[.]1/static/encrypt.min.js | url | | OTX | True | 2 | {‘pulse_count’: 3, ‘names’: [‘Underminer EK’ | 2022-12-15 01:55:15.135136+00:00 VirusTotal | hXXp://38[.]75[.]37[.]1/static/encrypt.min.js | url | | VirusTotal | False | 0 | Request forbidden. Allowed query rate may ha | 2022-12-15 01:55:15.135136+00:00 XForce | hXXp://38[.]75[.]37[.]1/static/encrypt.min.js | url | | XForce

You can also create dynamic summaries from a DataFrame and append DataFrame records to an existing dynamic summary.

Read the full documentation in MSTICPy Sentinel Dynamic Summaries doc

New QueryProvider API to dynamically add a parameterized query.

MSTICPy has always supported the ability to run ad hoc text queries for different providers and return the results as a DataFrame. Using a static query string like this is quick and easy if you only want to run a query once but what if you want to re-run with different time range or host name? A lot of tedious editing or string search/replace!

Adding a full query template to MSTICPy, on the other hand, is overkill for this kind of thing. Dynamic parameterized queries are especially suited for notebooks - you can create an in-line parameterized query and have it update with the new parameters every time you run the notebook.

To use dynamic queries - define the query with parameter placeholders (delimited with curly braces "{" and "}"), then create parameter objects (these handle any special formatting for datetimes, lists, etc.). You add the list of parameter objects along with the replaceable parameter values when you run the query, as shown below.

```python

intialize a query provider

qry_prov = mp.QueryProvider("MSSentinel")

define a query

query = """ SecurityEvent | where EventID == {eventid} | where TimeGenerated between (datetime({start}) .. datetime({end})) | where Computer has "{hostname}" """

define the query parameters

qphost = qryprov.Param("hostname", "str", "Name of Host") qpstart = qryprov.Param("start", "datetime") qpend = qryprov.Param("end", "datetime") qpevt = qryprov.Param("eventid", "int", None, 4688)

add the query

qryprov.addcustomquery( name="gethostevents", query=query, family="Custom", parameters=[qphost, qpstart, qpend, qp_evt] )

query is now available as

qryprov.Custom.gethostevents(hostname="MyPC"....) ``` See Dynamically Adding Queries in MSTICPy Docs

QueryProvider - Query Search

As the number of queries for some providers grows, it has become more difficult to quickly find the right query. We've implemented a simple search capability that lets you search over the names or properties of queries. It takes four parameters:

search - search terms to look for in the query name, description, parameter names, table and query text.
table - search terms to match on the target table of the query. (note: not all queries have the table parameter defined in their metadata)
param - search terms to match on a parameter name
case - boolean to force case-sensitive matching (default is case-sensitive).

The first three parameters can be a simple string or an iterable (e.g. list, tuple) of search terms. The search terms are treated as regular expressions. This means that a the search terms are treated as substrings (if no other regular expression syntax is included).

Find all queries that have the term "syslog" in their properties

python qry_prov.search("syslog") # equivalent to qry_prov.search(search="syslog") ['LinuxSyslog.all_syslog', 'LinuxSyslog.cron_activity', 'LinuxSyslog.list_account_logon_failures', ... See Search queries in MSTICPY Docs

Support for JSON queries in Data Providers

@FlorianBracq has updated the CyberReason data provider so that it supports JSON queries. The mechanism that we used for KQL and SQL queries breaks JSON since it is a simple string substitution. Other data sources that use JSON queries include Elastic - we are planning to leverage the same mechanism to support parameterized Elastic queries in a future release. Thanks @FlorianBracq!

What Else has Changed?

Kql query formatting by @FlorianBracq in https://github.com/microsoft/msticpy/pull/595
Fix minor linting issues in main by @petebryan in https://github.com/microsoft/msticpy/pull/604
Updated M365D and MDE data connectors with correct scopes when using delegated auth. by @petebryan in https://github.com/microsoft/msticpy/pull/580
Ianhelle/remove extranous nb 2022 11 28 by @ianhelle in https://github.com/microsoft/msticpy/pull/588
Enable native JSON support for Data Providers + move Cybereason driver to native JSON by @FlorianBracq in https://github.com/microsoft/msticpy/pull/584
Adding query search to data_providers.py by @ianhelle in https://github.com/microsoft/msticpy/pull/587
Fix typo by @FlorianBracq in https://github.com/microsoft/msticpy/pull/606
Ianhelle/mypy cache 2023 01 17 by @ianhelle in https://github.com/microsoft/msticpy/pull/608
Added API to QueryProvider to add a custom query at runtime by @ianhelle in https://github.com/microsoft/msticpy/pull/586
Bump sphinx from 5.3.0 to 6.1.3 by @dependabot in https://github.com/microsoft/msticpy/pull/605
Bump httpx from 0.23.0 to 0.23.3 by @dependabot in https://github.com/microsoft/msticpy/pull/607
Dynamic Summaries Sentinel API and DynamicSummary class. by @ianhelle in https://github.com/microsoft/msticpy/pull/593
Update sentinelanalytics.py listalert_rules API version. by @pensivepaddle in https://github.com/microsoft/msticpy/pull/592

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.2.0...v2.3.0

- Python
Published by ianhelle over 3 years ago

msticpy - IoC Defanging, ServiceNow, GCC support for MDE, Python 3.11

Highlights

Re-architected context and TI providers

The biggest feature of this release is not directly visible but has involved a huge amount of work by @FlorianBracq. Florian spotted that our HTTP TI provider (used for several TI services such as VirusTotal, OTX, XForce) could be used more generically, specifically for non-TI sources that provided valuable context, such as ServiceNow. So, he re-worked the TI providers sub-package to pull out generic context provider capabilities used by both TI and non-TI sources. The immediate benefit of this is the next highlight

ServiceNow context provider

This is yet to be full documented but if you have a ServiceNow instance and want to hook up MSTICPy to query it try the following. 1. Add your ServiceNow configuration to msticpyconfig.yaml yaml ContextProviders: ServiceNow: Primary: True Args: TenantId: 8360dd21-0294-4240-9128-89611f415c53 AuthKey: "authkey" AuthId: "authid" Provider: "ServiceNow" Note: you can store the secrets in KeyVault in the same way as TI and other Providers - see the Key Vault Secrets section of MSTICPy Settings Editor

Import and instantiate a ContextProvider and look things up ```python from msticpy.context.contextlookup import ContextLookup

contextlookup = ContextLookup() result = contextlookup.lookupobservable("10.0.0.1", providers=["ServiceNow"]) result2 = contextlookup.lookup_observable("user@some.dom", providers=["ServiceNow"]) ```

Defanging support for IoCExtract and TI Providers

In threat reports, IoCs are often de-fanged to make IP addresses, URLs, etc, not clickable. An example de-fanged IP address would look something like this 17[.]34[.]21[.]195

Previously these would not be matched by the IoCExtract patterns due to the "escaped" dots. IoCExtract now supports common de-fanged markup such as * "[.]" to escape dots in IP addresses and domains, * "@" replaced by "AT" * "http(s)" and "(s)ftp(s)" replaced by "hXXp(s)" and "(s)fXp(s)" respectively.

We have also added support for email address patterns to IoCExtract.

TI providers will also accept de-fanged IoCs, removing the de-fanging before submitting them to the provider for lookup.

We've also supplied a couple of utility functions defang_ioc and refang_ioc in msticpy.common.utility. These are not yet added as Pivot functions to IpAddress, Url, Dns, Account but will be added in a future release.

Added GCC support to MDE/M365 data providers

This allows customers working with government clouds to query the correct Defender endpoints.

Python 3.11 officially supported

Although there wasn't anything in our code that was a Py 3.11 blocker, some of our dependencies took a little while to publish 3.11-compatible wheels. That was all done with SciPy, Statsmodels and ScikitLearn and our build pipeline now in includes a full test pass on Python 3.11. Many thanks to @tonybaloney for pushing us through this.

What's Changed

Add base for Context Providers by @FlorianBracq in https://github.com/microsoft/msticpy/pull/511
Adding skip and warning to testvtpivot.py by @ianhelle in https://github.com/microsoft/msticpy/pull/560
Improved bug template getting rid of irrelevant sections by @ianhelle in https://github.com/microsoft/msticpy/pull/559
Intsights endpoint update. by @FlorianBracq in https://github.com/microsoft/msticpy/pull/526
Added support for GCC and Regional Clouds to MDE driver by @petebryan in https://github.com/microsoft/msticpy/pull/525
Resourcegraph - Incomplete list returned by @pensivepaddle in https://github.com/microsoft/msticpy/pull/496
Bump sphinx-rtd-theme from 1.0.0 to 1.1.0 by @dependabot in https://github.com/microsoft/msticpy/pull/553
Sumologic driver: custom dtypes options+fix, add paging, remove days duration int casting by @juju4 in https://github.com/microsoft/msticpy/pull/481
New mypy failures in kqlbase, elasticdriver, splunk_driver, sumolog… by @ianhelle in https://github.com/microsoft/msticpy/pull/564
Bump sphinx-rtd-theme from 1.1.0 to 1.1.1 by @dependabot in https://github.com/microsoft/msticpy/pull/563
Add 3.11 to test matrix by @tonybaloney in https://github.com/microsoft/msticpy/pull/546
Update dnspython requirement from <=2.0.0 to <3.0.0 by @dependabot in https://github.com/microsoft/msticpy/pull/289
Inability to fetch "all" incidents, only 50 by @pensivepaddle in https://github.com/microsoft/msticpy/pull/565
Add de-fanging support for iocextract and TI providers by @ianhelle in https://github.com/microsoft/msticpy/pull/536
Implementing isort for context classes, adding missing docs by @ianhelle in https://github.com/microsoft/msticpy/pull/567
Add support for context provider Service Now by @FlorianBracq in https://github.com/microsoft/msticpy/pull/556
Added Sentinel TI integration features. by @petebryan in https://github.com/microsoft/msticpy/pull/532
Ianhelle/pygeohash and exceptions 2022 11 11 by @ianhelle in https://github.com/microsoft/msticpy/pull/566
Removing debug prints and duplicate code. by @petebryan in https://github.com/microsoft/msticpy/pull/570
Moving ASN http lookup to execute at runtime, when whois lookup happens. by @ianhelle in https://github.com/microsoft/msticpy/pull/568
Added a new set of Sentinel queries related to network activity using the CommonSecurityLog data source. by @petebryan in https://github.com/microsoft/msticpy/pull/524
Fixed issues with dataprovider instances by @ianhelle in https://github.com/microsoft/msticpy/pull/549
Adding AzureAuthentication.rst by @ianhelle in https://github.com/microsoft/msticpy/pull/578

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.1.5...v2.2.0

- Python
Published by ianhelle over 3 years ago

msticpy - Bokeh, ipywidgets version restrictions

The main driver for this release is to restrict versions of bokeh, ipywidgets and pandas. * Version 3.0.0 of bokeh plots has some breaking changes that prevent it working with MSTICPy * Version 8.0.0 of ipywidgets has changes that prevent some of the MSTICPy compound widgets displaying correctly.

We also decided to start restricting versions of some of our other dependencies to the current major version - to prevent unexpected breaking changes stopping MSTICPy from working. We have included pandas in this list and will expand it to cover more packages in future. We will combine this with an automated build job that has no version restrictions so that we're aware of version changes that we need to address. The intent here is to have MSTICPy have as broad a version range as possible for its dependencies while still avoiding failures due to breaking changes.

Another small but important change is an update to the Process Tree viewer to allow process GUIDs as process IDs (rather than just hex or decimal format integers). Thanks to @nbareil for this change!

What's Changed

process_tree: Accept GUID format for ProcessID and ParentProcessID by @nbareil in https://github.com/microsoft/msticpy/pull/542
Bump sphinx from 5.1.1 to 5.3.0 by @dependabot in https://github.com/microsoft/msticpy/pull/540
Bump readthedocs-sphinx-ext from 2.1.9 to 2.2.0 by @dependabot in https://github.com/microsoft/msticpy/pull/545
Update AzureBlobStorage.rst by @garybushey in https://github.com/microsoft/msticpy/pull/539
Adding upper version restrictions to bokeh, pandas and ipywidgets deps by @ianhelle in https://github.com/microsoft/msticpy/pull/552

New Contributors

@garybushey made their first contribution in https://github.com/microsoft/msticpy/pull/539

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.1.4...v2.1.5

- Python
Published by ianhelle over 3 years ago

msticpy - Fixes for MS Sentinel API and configuration

Some minor fixes and improvements:

MicrosoftSentinel class now defaults to "Default" workspace or workspace name supplied as workspace parameter when connecting. python sentinel = MicrosoftSentinel() sentinel.connect() # connect to "Default" workspace sentinel.connect(workspace="MyWorkspace") # connect to named workspace
Sentinel create_* APIs now return ID of new item (incident, bookmark, analytic, watchlist)
initnotebook - now accepts config parameter to use custom msticpyconfig.yaml for notebook session (overrides enviromnent variable and other defaults ```python import msticpy as mp mp.initnotebook(config="~/configs/alltiprovs.yaml") # use a custom msticpy config file. ```
Sentinel configuration editor no longer throws an exception if named control not found
Sentinel TI provider will not attempt lookups if ThreatIntelligenceIndicator table not found in the Sentinel data provider schema
Support for Kusto/Azure Data explorer settings in Settings editor
Added checked_kwargs decorator to utility/types.py

What's Changed

Ianhelle/training hotfixes 2022 10 13 by @ianhelle in https://github.com/microsoft/msticpy/pull/543
Updated ReadMe with Blackhat Arsenal Tag by @petebryan in https://github.com/microsoft/msticpy/pull/521

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.1.3...v2.1.4

- Python
Published by ianhelle over 3 years ago

msticpy - Process Tree Viewer updates

Highlights

This is a minor release with some fixes and additions that enable broader functionality. The biggest-impacting changes apply to the Process Tree visualization. These changes allow it to work with broader types of Windows or Linux process data:

Removed the following columns that were previously required: hostname, logonid, username, cmdline.
Added auto-coloring by level if no legend is supplied.
Fixed process sorting so that tree and peer groups in the tree are sorted by level, then timestamp.
Added ability to supply schema as dictionary to the process tree APIs.

The changes are described in more detail below.

We've also added support for a new MS Sentinel API to retrieve queries stored in a Sentinel workspace and fixed some issues in IP WhoIs lookups.

Process Tree changes

Reduced required column set

This allows you to use the process tree visualization and utilities with a minimal set of data fields: * processid * parentid * processname * timestamp

```python

cust_schema = {
    "process_name": "ImageFileName",
    "process_id": "PID",
    "parent_id": "PPID",
    "time_stamp": "CreateTime",
}
df.mp_plot_process(schema=cust_schema)

```

Auto-coloring of tree plot

If you do not supply a legend_col parameter, the process objects will be automatically colored by level in the hierarchy. This makes a basic tree more colorful and easier to navigate. Proctree-default-color

Processes are correctly sorted by process time

Previously, the code that builds the process tree left individual processes in an unintuitive order. For a given level (e.g. parents) all of the processes will be displayed in time created order.

For example: A \ - A.1 - A.2 B \ - B.1 - B.2 A will always have a timestamp less than or equal to B. All children of A (A.1, A.2...) and B will be shown in time created order. However, across different levels and peer groups, there is no guarantee of time-ordering. In the above example, even though timestamp A is less than timestamp B, B.1 and B.2 could have timestamps earlier than either A.1 or A.2.

	path	ImageFileName	CreateTime
proc_key
registry\|88\|2021-04-01 05:04:54.000000	116/0	Registry	2021-04-01 05:04:54+00:00
system\|4\|2021-04-01 05:04:58.000000	117/1	System	2021-04-01 05:04:58+00:00
smss.exe\|404\|2021-04-01 05:04:58.000000	117/1/2	smss.exe	2021-04-01 05:04:58+00:00
csrss.exe\|640\|2021-04-01 05:05:00.000000	118/3	csrss.exe	2021-04-01 05:05:00+00:00
winlogon.exe\|700\|2021-04-01 05:05:00.000000	118/4	winlogon.exe	2021-04-01 05:05:00+00:00
dwm.exe\|1028\|2021-04-01 05:05:02.000000	118/4/17	dwm.exe	2021-04-01 05:05:02+00:00
logonui.exe\|512\|2021-04-01 05:05:02.000000	118/4/21	LogonUI.exe	2021-04-01 05:05:02+00:00
fontdrvhost.ex\|960\|2021-04-01 05:05:01.000000	118/4/7	fontdrvhost.ex	2021-04-01 05:05:01+00:00
wininit.exe\|632\|2021-04-01 05:05:00.000000	119/5	wininit.exe	2021-04-01 05:05:00+00:00
lsass.exe\|776\|2021-04-01 05:05:01.000000	119/5/10	lsass.exe	2021-04-01 05:05:01+00:00

mpplot.processtree and mp.buildprocesstree support schema as dictionary

Previously these accessors and the underlying functions plot_process_tree and build_process_tree would only accept msticpy.transform.process_tree_schema.ProcSchema instances. These will now accept dictionaries with at least the minimum required attributes as keys.

What's Changed

Sentinel - Return all saved queries by @petebryan in https://github.com/microsoft/msticpy/pull/519
Bump readthedocs-sphinx-ext from 2.1.8 to 2.1.9 by @dependabot in https://github.com/microsoft/msticpy/pull/507
Bump respx from 0.19.2 to 0.20.0 by @dependabot in https://github.com/microsoft/msticpy/pull/512
Allow process tree to work with more data sources. by @ianhelle in https://github.com/microsoft/msticpy/pull/513
Fixed error in cell using non-existing column name by @ianhelle in https://github.com/microsoft/msticpy/pull/527
Ianhelle/proc tree fixes 2022 09 16 by @ianhelle in https://github.com/microsoft/msticpy/pull/530
Fixed issue with whois lookups on only local IPs by @petebryan in https://github.com/microsoft/msticpy/pull/506

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.1.2...v2.1.3

- Python
Published by ianhelle over 3 years ago

msticpy - Hotfix - Azure authentication failure

A last-minute change before release of 2.1.0 introduced a critical bug in azureauthcore.py. This caused all azure authentication to fail. It would also cause init_notebook() to fail if the user had any Key Vault secrets referenced in their msticpyconfig.yaml.

Thanks to @FlorianBracq for spotting this independently (and before us) and submitting a PR with the fix. The PR below is essentially the same fix as Florian's with a subtle change to allow an EnvironmentCredential of None to appear in the list of creds sent to ChainedTokenCredential. This is to cover an edge case where EnvironmentCredential is requested but the required environment variables are not set.

What's Changed

[fix] bug in call to ChainTokenCredential breaks all authentication by @ianhelle in https://github.com/microsoft/msticpy/pull/505
Rolling back change on buildchained_creds by @FlorianBracq in https://github.com/microsoft/msticpy/pull/504

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.1.1...v2.1.2

- Python
Published by ianhelle almost 4 years ago

msticpy - Hotfix - missing beautifulsoup4 from requirements/dependencies

We inadvertently took a hard dependency on beautifulsoup4 but didn't have it in our dependencies. Unfortunately, since bs4 is in our test dependencies this passed all the tests so didn't spot it until later.

What's Changed

Add bs4 to requirements by @ianhelle in https://github.com/microsoft/msticpy/pull/501

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.1.0...v2.1.1

- Python
Published by ianhelle almost 4 years ago

msticpy - IpWhois, Malware Bazaar, Azure Auth, Azure Synapse

Highlights

Replaced dependency on IPWhois with local code #479

The ipwhois package seems to be abandoned and was causing conflicts with dnspython. We've created equivalent functionality in msticpy removing build warnings and (minutely) speeding up install time. We've also added a MSTICPy pandas accessor df.mp.whois() so that you can do bulk queries from a dataframe.

Malware Bazaar TI Provider #459

Many thanks to @fr0gger for this. Check out the notebook MBLookup to see how you can use this new provider.

Documentation on how to build a Data Provider #465

This was previously a blog post but we've added it to the official docs - Writing and Contributing a Data Provider

Updates to Azure authentication to support more authentication types #484

We've switched from using DefaultAzureCredential to supporting the native credential types. This lets us support additional credential types such as Client Secret and Certification authentication. You can also create your own custom AzureCredential and pass this to az_connect.

Updates to SQL2Kql converter #488

This was really prompted by @tonybaloney in helping us get a build working on Python 3.8-3.11. This module had a dependency on a now-deprecated mozsqlparser. We've updated to use mosqlparsing - many thanks to @klahnakoski for work on keeping this alive and well. Our module also contains some fixes and enhancements from the original.

Builds and tests now running on Python 3.8, 3.9, 3.10 #476

We were previously only building on Python 3.8. Huge thanks to @tonybaloney for working on this and bringing us into the modern era. We still have some issues with Python 3.11dev - although this is due to SciPy breaking with the last 3.11 version we tried. As soon as this is sorted we will add 3.11 back.

Added support for msticpy notebooks in Azure Synapse pipelines #493

This is mostly work done to support MS Sentinel running unattended notebooks in Synapse pipelines. We've extended the mp.init_notebook() function so that it can correctly configure msticpy (looking for msticpyconfig.yaml on a mounted blob storage container and persisting cached data there), use the linked Key Vault to store secrets and supply service principal credentials to msticpy.

Important fixes

Allow for missing columns in Folium map data frame #489
Updated M365D/MDE driver to pass query request with JSON encoding #498 Defender have always supported this but we were sending a JSON string, which they recently stopped supporting. This should be working again.
You can now see data query help before connecting to the data provider. It's also possible to dry run the query to see the full query with parameters replaced without needing to connect. #482

What's Changed

Azure ML init fix by @FlorianBracq in https://github.com/microsoft/msticpy/pull/471
Sumologic-DataConnector.ipynb: fix s/url=/connection_str=/ by @juju4 in https://github.com/microsoft/msticpy/pull/436
Fix typo in parameter name by @FlorianBracq in https://github.com/microsoft/msticpy/pull/470
Update jinja2 requirement from <3.1.0 to <3.2.0 by @dependabot in https://github.com/microsoft/msticpy/pull/450
Update docutils requirement from <0.18.0 to <0.20.0 by @dependabot in https://github.com/microsoft/msticpy/pull/464
Ianhelle/writing data provs doc 2022 03 14 by @ianhelle in https://github.com/microsoft/msticpy/pull/465
IP Whois and Azure Auth Fixes by @petebryan in https://github.com/microsoft/msticpy/pull/479
Bump sphinx from 5.0.2 to 5.1.1 by @dependabot in https://github.com/microsoft/msticpy/pull/478
Update UploadData.rst with fix for import by @pensivepaddle in https://github.com/microsoft/msticpy/pull/483
[update] DNS resolver return dataframe with one IP per row by @ianhelle in https://github.com/microsoft/msticpy/pull/485
Adding Malware Bazaar module as TI provider by @fr0gger in https://github.com/microsoft/msticpy/pull/459
Allow query help when qry provider not connected by @ianhelle in https://github.com/microsoft/msticpy/pull/482
Adding all supported auth types to config UI mpconfig_defaults.yaml by @ianhelle in https://github.com/microsoft/msticpy/pull/484
Fixing build issues with requirements-dev and doc by @ianhelle in https://github.com/microsoft/msticpy/pull/490
[fix] Updated SQL to KQL converter to use mosqlparsing by @ianhelle in https://github.com/microsoft/msticpy/pull/488
[fix] Allow for missing columns in Folium GeoIP data. by @ianhelle in https://github.com/microsoft/msticpy/pull/489
Add support for Python 3.10, 3.11 and test in CI by @tonybaloney in https://github.com/microsoft/msticpy/pull/476
Revert "Add support for Python 3.10, 3.11 and test in CI" by @petebryan in https://github.com/microsoft/msticpy/pull/494
Add support for Python 3.10, 3.11 and test in CI by @petebryan in https://github.com/microsoft/msticpy/pull/495
Added azuresynapsetools to support notebooks in Synapse by @ianhelle in https://github.com/microsoft/msticpy/pull/493
Changing the MDE/M365D request content to json encoding. by @ianhelle in https://github.com/microsoft/msticpy/pull/498
Fixes and updates to support notebooklet updates by @petebryan in https://github.com/microsoft/msticpy/pull/497
Fix breaking issues in Auth and Browshot by @petebryan in https://github.com/microsoft/msticpy/pull/499

New Contributors

@juju4 made their first contribution in https://github.com/microsoft/msticpy/pull/436
@tonybaloney made their first contribution in https://github.com/microsoft/msticpy/pull/476

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.0.0...v2.1.0

- Python
Published by ianhelle almost 4 years ago

msticpy - MSTICPy Version 2.0

MSTICPy Release 2.0

A notebook containing some of the features of MSTICPy 2.0 is available at What's new in MSTICPy 2.0

If you are new to MSTICPy or just want to catch up and get a quick overview check out our new MSTICPy Quickstart Guide.

Dropping Python 3.6 support
Package re-organization and module search
Simplifying imports in MSTICPy
Folium map update - single function, layers, custom icons
Threat Intelligence providers - async support
Time Series simplified - analysis and plotting
DataFrame to graph/network visualization
Pivots - easy initialization/dynamic data pivots
Consolidating Pandas accessors
MS Sentinel workspace configuration
MS Defender queries available in MS Sentinel QueryProvider
Microsoft Sentinel QueryProvider
New queries
Documentation Additions and Improvements
Miscellaneous improvements
Previous feature changes since MSTICPy 1.0

Dropping Python 3.6 support

As of this release we only officially support Python 3.8 and above. We will try to support Python 3.6 if the fixes required are small and contained but make no guarantees of it working completely on Python prior to 3.8.

Package re-organization and module search

One of our main goals for V2.0.0 was to re-organize MSTICPy to be more logical and easier to use and maintain. Several years of organic growth had seen modules created in places that seemed like a good idea at the time but did not age well.

The discussion about the V2 structure can be found here #320.

Due to the re-organization, many features are no longer in places where they used to be imported from!

We have tried to maintain compatibility with old locations by adding "glue" modules. These allow import of many modules from their previous locations but will issue a Deprecation warning if loaded from the old location. The warning will contain the new location of the module - so you should update your code to point to this new location.

This table gives a quick overview of the V2.0 structure

| folder | description | |-----------|----------------------------------------------------------------------------------| | analysis | data analysis functions - timeseries, anomalies, clustering | | auth | authentication and secrets management | | common | common used utilities and definitions (e.g. exceptions) | | config | configuration and settings UI | | context | enrichment modules geoip, ip_utils, domaintools, tiproviders, vtlookup | | data | data acquisition/queries/storage/uploaders | | datamodel | entities, soc objects | | init | package loading and initialization - nbinit, pivot modules | | nbwidgets | nb widgets modules | | transform | simple data processing - decoding, reformatting, schema change, process tree | | vis | visualization modules including browsers |

Notable things that have moved:

most things from the sectools folder have migrated to context, transform or analysis
most things from the nbtools folder have migrated to:
- msticpy.init - (not to be confused with __init__) - package initialization
- msticpy.vis - visualization modules
pivot functionality has moved to msticpy.init

Module Search

If you are having trouble finding a module, we have added a simple search function:

python import msticpy as mp mp.search("riskiq")

Matches will be returned in a table with links to the module documentation

Modules matching 'riskiq'

Module	Help
msticpy.context.tiproviders.riskiq	msticpy.context.tiproviders.riskiq

Simplifying imports in MSTICPy

The root module in MSTICPy now has several modules and classes that can be directly accessed from it (rather than having to import them individually).

We've also decided to adopt a new "house style" of importing msticpy as the alias mp. Slavishly copying the idea from some of admired packages that we use (pandas -> pd, numpy -> np, networkx -> nx) we thought it would save a bit of typing. You are free to adopt or ignore this style - it obviously has no impact on the functionality.

python import msticpy as mp mp.init_notebook() qry_prov = mp.QueryProvider("MDE") ti = mp.TILookup()

Many commonly-used classes and functions are exposed as attributes of msticpy (or mp).

Also a number of commonly-used classes are imported by default by init_notebook, notably all of the entity classes. This makes it easier to use pivot functions without any initialization or import steps.

```python import msticpy as mp mp.init_notebook()

# IpAddress can be used without having to import it.
IpAddress.whois("123.45.6.78")

```

`init_notebook` improvements

You no longer need to supply the namespace=globals() parameter when calling from a notebook. init_notebook will automatically obtain the notebook global namespace and populate imports into it.
The default verbosity of init_notebook is now 0, which produces minimal output - use verbosity=1 or verbosity=2 to get more detailed reporting.
The Pivot subsystem is automatically initialized in init_notebook.
All MSTICPy entities are imported automatically.
All MSTICPy magics are initialized here.
Most MSTICPy pandas accessors are initialized here (some, which require optional packages, such as the timeseries accessors are not initialized by default).
init_notebook supports a config parameter - you can use this to provide a custom path to a msticpyconfig.yaml overriding the usual defaults.
searching for a config.json file is only enabled if you are running MSTICPy in Azure Machine Learning.

Folium map update - single function, layers, custom icons

The Folium module in MSTICPy has always been a bit complex to use since it normally required that you convert IP addresses to MSTICPy IpAddress entities before adding them to the map.

You can now plot maps with a single function call from a DataFrame containing IP addresses or location coordinates. You can group the data into folium layers, specify columns to populate popups and tooltips and to customize the icons and coloring.

folium_layers

plot_map

A new plot_map function (in the msticpy.vis.foliummap module) that lets you plot mapping points directly from a DataFrame. You can specify either an ip_column or coordinates columns (lat_column and long_column). In the former case, the geo location of the IP address is looked up using the MaxMind GeoLiteLookup data.

You can also control the icons used for each marker with the icon_column parameters. If you happen to have a column in your data that contains names of FontAwesome or GlyphIcons icons you can use that column directly. More typically, you would combine the icon_column with the icon_map parameter. You can specify either a dictionary or a function. For a dictionary, the value of the row in icon_column is used as a key - the value is a dictionary of icon parameters passed to the Folium.Icon class. For a function, the icon_column value is passed to the function as a single parameter and the return value should be a dictionary of valid parameters for the Icon class. You can read the documentation for this function in the docs

plot_map pandas accessor

Plot maps from the comfort of your own DataFrame! Using the msticpy mp_plot accessor you can plot maps directly from a DataFrame containing IP or location information. The folium_map function has the same syntax as plot_map except that you omit the data parameter.

```python

df.mp_plot.folium_map(ip_column="ip", layer_column="CountryName")

```

pd_accessors

Layering, Tooltips and Clustering support

In plot_map and .mp_plot.folium_map you can specify a layer_column parameter. This will group the data by the values in that column and create an individually selectable/displayable layer in Folium. For performance and sanity reasons this should be a column with a relatively small number of discrete values.

Clustering of markers in the same layer is also implemented by default - this will collapse multiple closely located markers into a cluster that you can expand by clicking or zooming.

You can also populate tooltips and popups with values from one or more column names.

"Classic" interface

The original FoliumMap class is still there for more manual control. This has also been enhanced to support direct plotting from IP, coordinates or GeoHash in addition to the existing IpAddress and GeoLocation entities. It also supports layering and clustering.

Threat Intelligence providers - async support

When you have configured more than one TI provider, MSTICPy will execute requests to each of them asynchronously. This will bring big performance benefits when querying IoCs from multiple providers. Note: requests to individual providers are still executed synchronously since we want to avoid swamping provider services with multiple simultaneous requests.

We've also implemented progress bar tracking for TILookups, giving a visual indication of progress when querying multiple IoCs.

Combining the progress tracking with asynchronous operation means that not only is performing lookups for lots of observables faster but you will also less likely to be left guessing whether or not your kernel has hung.

Note that asynchronous execution only works with lookup_iocs and TI lookups done via the pivot functions. lookup_ioc will run queries to multiple providers in seqence so will usually be a lot slower than lookup_iocs.

python # don't do this ti_lookup.lookup_ioc("12.34.56.78") # do this (put a single IoC in a list) ti_lookup.lookup_iocs(["12.34.56.78"])

TI Providers are now also loaded on demand - i.e. only when you have a configuration entry in your msticpyconfig.yaml for that provider. This prevents loading of code (and possibly import errors) due to providers which you are not intending to use.

Finally, we've added functions to enable and disable providers after loading TILookup:

```python

from msticpy.context import TILookup
ti_lookup = TILookup()

iocs = ['162.244.80.235', '185.141.63.120', '82.118.21.1', '85.93.88.165']
ti_lookup.lookup_iocs(iocs, providers=["OTX", "RiskIQ"])

```

ti_providers_async

Time Series simplified - analysis and plotting

Although the Time Series functionality was relatively simple to use, it previously required several disconnected steps to compute the time series, plot the data, extract the anomaly periods. Each of these needed a separate function import. Now you can do all of these from a DataFrame via pandas accessors. (currently there is a separate accessor df.mp_timeseries but we are still working on consolidating our pandas accessors so this may change before the final release.)

Because you typically still need these separate outputs, the accessor has multiple methods:

df.mp_timeseries.analyze - takes a time-summarized DataFrame and returns the results of a time-series decomposition
df.mp_timeseries.plot - takes a decomposed time-series and plots the anomalies
df.mp_timeseries.anomaly_periods - extracts anomaly periods as a list of time ranges
df.mp_timeseries.anomaly_periods - extracts anomaly periods as a list of KQL query clauses
df.mp_timeseries.apply_threshold - applies a new anomaly threshold score and returns the results.

See documentation

Analyze data to produce time series.

```python

df = qry_prov.get_networkbytes_per_hour(...)
ts_data = df.mp_timeseries.analyze()

```

Analyze and plot time series anomalies

```python

df = qry_prov.get_networkbytes_per_hour(...)
ts_data = df.mp_timeseries.analyze().mp_timeseries.plot()

```

Time series plot

Analyze and retrieve anomaly time ranges

```python

df = qry_prov.get_networkbytes_per_hour(...)
df.mp_timeseries.analyze().mp_timeseries.anomaly_periods()

```

raw [TimeSpan(start=2019-05-13 16:00:00+00:00, end=2019-05-13 18:00:00+00:00, period=0 days 02:00:00), TimeSpan(start=2019-05-17 20:00:00+00:00, end=2019-05-17 22:00:00+00:00, period=0 days 02:00:00), TimeSpan(start=2019-05-26 04:00:00+00:00, end=2019-05-26 06:00:00+00:00, period=0 days 02:00:00)]

```python

df = qry_prov.get_networkbytes_per_hour(...)
df.mp_timeseries.analyze().mp_timeseries.kql_periods()

```

raw '| where TimeGenerated between (datetime(2019-05-13 16:00:00+00:00) .. datetime(2019-05-13 18:00:00+00:00)) or TimeGenerated between (datetime(2019-05-17 20:00:00+00:00) .. datetime(2019-05-17 22:00:00+00:00)) or TimeGenerated between (datetime(2019-05-26 04:00:00+00:00) .. datetime(2019-05-26 06:00:00+00:00))'

DataFrame to graph/network visualization

You can convert a pandas DataFrame into a NetworkX graph or plot directly as a graph using Bokeh interactive plotting.

You pass the functions the column names for the source and target nodes to build a basic graph. You can also name other columns to be node or edge attributes. When displayed these attributes are visible as popup details courtesy of Bokeh’s Hover tool.

python proc_df.head(100).mp_plot.network( source_col="SubjectUserName", target_col="Process", source_attrs=["SubjectDomainName", "SubjectLogonId"], target_attrs=["NewProcessName", "ParentProcessName", "CommandLine"], edge_attrs=["TimeGenerated"], )

Graph plot

Pivots - easy initialization/dynamic data pivots

The pivot functionality has been overhauled. It is now initialized automatically in init_notebook - so you don't have to import and create an instance of Pivot.

Better data provider support

Previously, queries from data providers were added at initialization of the Pivot subsystem. This meant that you had to:

create your query providers before starting Pivot
every time you created a new QueryProvider instance you had to re-initialize Pivot.

Data providers now dynamically add relevant queries as pivot functions when you authenticate.

Multi-instance provider support

Some query providers (such as MS Sentinel) support multiple instances. Previously this was not well supported in Pivot functions - the last provider loaded would overwrite the queries from earlier providers. Pivot now supports separate instance naming so that each Workspace has a separate instance of a given pivot query.

Threat Intelligence pivot functions

The naming of the Threat Intelligence pivot functions has been simplified considerably.

VirusTotal and RiskIQ relationship queries should now be available as pivot functions (you need the VT 3 and PassiveTotal packages installed respectively to enable this functionality).

More Defender query pivots

A number of MS Defender queries (using either the MDE or MSSentinel QueryProviders) are exposed as Pivot functions.

Consolidating Pandas accessors

Pandas accessors let you extend a pandas DataFrame or Series with custom functions. We use these in MSTICPy to let you call analysis or visualization functions as methods of a DataFrame.

Most of the functions previously exposed as pandas accessors, plus some new ones, have been consolidated into two main accessors.

df.mp - contains all of the transformation functions like base64 decoding, ioc searching, etc.
df.mp_plot - contains all of the visualization accessors (timeline, process tree, etc.)

`mp` accessor

b64extract - base64/zip/gzip decoder
buildprocesstree - build process tree from events
ioc_extract - extract observables by pattern such as IPs, URLs, etc.
mask - obfuscate data to hide PII
to_graph - transform to NetworkX graph

`mp_plot` accessor

folium_map - plot a Folium map from IP or coordinates
incident_graph - plot an incident graph
matrix - plot correlation between two values
network - plot graph/network from tabular data
process_tree - plot process tree from process events
timeline - plot a multi-grouped timeline of events
timeline_duration - plot grouped start/end of event sequence
timeline_values - plot timeline with a scalar values

Example usage (note: the required parameters, if any, are not shown)

```python

df.mp.ioc_extract(...)
df.mp.to_graph(...)
df.mp.mask(...)

df.mp_plot.timeline(...)
df.mp_plot.timeline_values(...)
df.mp_plot.process_tree(...)
df.mp_plot.network(...)
df.mp_plot.folium_map(...)

```

One of the benefits of using accessors is the ability to chain them into a single pandas expression (mixing with other pandas methods).

```python

(
    my_df
    .mp.ioc_extract(...)
    .groupby(["IoCType"])
    .count()
    .reset_index()
    .mp_plot.timeline(...)
)

```

MS Sentinel workspace configuration

From MPConfig edit you can more easily import and update your Sentinel workspace configuration.

Config Editor showing import URL and Resolve Settings buttons

Resolve Settings

If you have a minimal configuration (e.g. just the Workspace ID and Tenant ID) you can retrieve other values such as Subscription ID, Workspace Name and Resource Group and save them to your configuration using the Resolve Settings button

Import Settings from URL

You can copy the URL from the Sentinel portal and paste it into the the MPConfigEdit interface. It will extract and lookup the full details of the workspace to save to your settings.

Expanded Sentinel API support

The functions used to implement the above functionality are also available standalone in the MSSentinel class.

```python from msticpy.context.azure import MicrosoftSentinel

MicrosoftSentinel.get_workspace_details_from_url(url)
MicrosoftSentinel.get_workspace_name(ws_id)
MicrosoftSentinel.get_workspace_settings(resource_id)
MicrosoftSentinel.get_workspace_settings_by_name(ws_name, sub_id, res_group)
MicrosoftSentinel.get_workspace_id(ws_name, sub_id, res_group)

```

MS Defender queries available in MS Sentinel QueryProvider

Since Sentinel now has the ability to import Microsoft Defender data, we've made the Defender queries usable from the MS Sentinel provider.

python qry_prov = QueryProvider("MSSentinel") qry_prov.MDE.list_host_processes(host_name="my_host")

This is a more general functionality that allows us to share compatible queries between different QueryProviders.

Many of the MS Defender queries are also now available as Pivot functions.

Microsoft Sentinel QueryProvider

The MS Sentinel provider now support a timeout parameter allowing you lengthen and shorten the default.

python qry_prov.MDE.list_host_processes( host_name="myhost", timeout=600, )

You can set other options supported by Kqlmagic when initializing the provider

python qry_prov = mp.QueryProvider("MSSentinel", cloud="government")

You can specify a workspace name as a parameter when connecting instead of creating a WorkSpaceConfig instance or supplying a connection string. To use the Default workspace supply "Default" as the workspace name.

python qry_prov.connect(workspace="MyWorkspace")

New queries

Several new Sentinel and MS Defender queries have been added.

| QueryGroup | Query | Description | |------------------|----------------------------------------|-----------------------------------------------------------| | AzureNetwork | networkconnectionstourl | List of network connections to a URL | | LinuxSyslog | notableevents | Returns all syslog activity for a host | | LinuxSyslog | summarizeevents | Returns a summary of activity for a host | | LinuxSyslog | sysmonprocessevents | Get Process Events from a specified host | | WindowsSecurity | accountchangeevents | Gets events related to account changes | | WindowsSecurity | listlogonattemptsbyip | Retrieves the logon events for an IP Address | | WindowsSecurity | notableevents | Get noteable Windows events not returned in other queries | | WindowsSecurity | schdldtasksandservices | Gets events related to scheduled tasks and services | | WindowsSecurity | summarizeevents | Summarizes a the events on a host |

Over 30 MS Defender queries can now also be used in MS Sentinel workspaces if MS Defender for Endpoint/MS Defender 365 data is connected to Sentinel

Additional Azure resource graph queries

| QueryGroup | Query | Description | |----------------|------------------------------------------|-----------------------------------------------------------------------------------------------------| | Sentinel | getsentinelworkspaceforresourceid | Retrieves Sentinel/Azure monitor workspace details by resource ID | | Sentinel | getsentinelworkspaceforworkspaceid | Retrieves Sentinel/Azure monitor workspace details by workspace ID | | Sentinel | listsentinelworkspacesforname | Retrieves Sentinel/Azure monitor workspace(s) details by name and resource group or subscription_id |

See the updated built-in query list

Documentation Additions and Improvements

The documentation for V2.0 is available at https://msticpy.readthedocs.io (Previous versions are still online and can be accessed through the ReadTheDocs interface).

New and updated documents

New MSTICPy Quickstart Guide
Updated Installing guide
Updated MSTICPy Package Configuration
Updated Threat Intel Lookup documentation
Updated Time Series analysis documentation
New Plot Network Graph from DataFrame
Updated Plotting Folium maps
Updated Pivot functions
Updated Jupyter and Sentinel

API documentation

As well as including all of the new APIs, the API documentation has been split into a module-per-page to make it easier to read and navigate.

InterSphinx

The API docs also now support "InterSphinx". This means that MSTICPy references to objects in other packages (e.g. Python standard library, pandas, Bokeh) have active links that will take you to the native documentation for that item.

Sample notebooks

The sample notebooks for most of these features have been updated along the same lines. See MSTICPy Sample notebooks

There are three new notebooks:

ContiLeaksAnalysis
Network Graph from DataFrame
What's new in MSTICPy 2.0

ContiLeaks notebook added to MSTICPy Repo

We are privileged to host Thomas's awesome ContiLeaks notebook that covers investigation into attacker forum chats including some very cool illustration of using natural language translation in a notebook.

Thanks @fr0gger!

Miscellaneous improvements

MSTICPy network requests use a custom User Agent header so that you can identify or track requests from MSTICPy/Notebooks.
GeoLiteLookup and the TOR and OpenPageRank providers no longer try to download data files at initialization - only on first use.
GeoLiteLookup tries to keep a single instance if the parameters that you initialize it with are the same
Warnings cleanup - we've done a lot of work to clean up warnings - especially deprecation warnings.
Moved some remaining Python unittest tests to pytest

Feedback

Please reach out to us on GitHub - file an issue or start a discussion on https://github.com/microsoft/msticpy - or msticpy@microsoft.com

Previous feature changes since MSTICPy 1.0

Sentinel Search API Support v1.8.0
Azure authentication improvements v1.8.0
Powershell deobfuscator and viewer v.1.7.5
Splunk Async queries v.1.7.5
CyberReason QueryProvider @FlorianBracq v1.7.0
IntSights TI provider @FlorianBracq v1.7.0
Splunk queries @d3vzer0 v1.7.0
Moved from requests to httpx @grantv9 v1.7.0
MS Sentinel API support for watchlists and analytics v1.6.0
Clustering, grouping and layering support for Folium @tj-senserva v1.6.0
Process Tree visualization supports multiple data schemas v1.6.0
VT FileBehavior, File object browser and Pivot functions v1.6.0
Single sign-on for notebooks in AML v1.5.1
RiskIQ TI Provider and Pivot functions @aeetos v1.5.1
Sentinel Incident and Entity graph exploration and visualization v1.5.0
Support for Azure Data Explore (Kusto) QueryProvider v1.5.0
Support for M365D QueryProvider v1.5.0
Added GitHub actions CI pipeline and updated Azure pipelines CI v1.5.0
Support for Azure sovereign clouds v1.4.0
Process Tree visualization for MDE data v1.4.0
Matrix plot visualization v1.4.0
Enable MSTICPy use from applications and scripts v.1.3.1
Timeline duration visualization v1.3.0
Azure Resource Graph provider @rcobb-scwx v1.2.1
Sumologic QueryProvider @juju4 v1.2.1
Notebook data viewer v1.2.0
Pivot functions updates - joins for all pivot types, shortcuts v1.1.0
GreyNoise TI Provider v1.1.0

- Python
Published by ianhelle almost 4 years ago

msticpy - MSTICPy 2.0 - Pre-release 3

New Features

A notebook containing some of the features of MSTICPy 2.0 is available at What's new in MSTICPy 2.0

Dropping Python 3.6 Support

As of this release we only officially support Python 3.8 and above. We will try to support Python 3.6 if the fixes required are small and contained but make no guarantees of it working completely on Python prior to 3.8

DataFrame to Graph/Network

You can convert a pandas DataFrame into a NetworkX graph or plot directly as a graph using Bokeh interactive plotting.

You pass the functions the column names for the source and target nodes to build a basic graph. You can also name other columns to be node or edge attributes. When displayed these attributes are visible as popup details courtesy of Bokeh’s Hover tool.

python proc_df.head(100).mp_plot.network( source_col="SubjectUserName", target_col="Process", source_attrs=["SubjectDomainName", "SubjectLogonId"], target_attrs=["NewProcessName", "ParentProcessName", "CommandLine"], edge_attrs=["TimeGenerated"], ) Graph plot

Pivots without initialization/dynamic data query import

The pivot functionality has been overhauled - it is now initialized automatically in init_notebook.

Previously queries from data providers were added at initialization - meaning that you had to create your query providers before starting pivot or re-initialize pivot. Data providers now dynamically add relevant queries as pivot functions when you authenticate. Also for some providers, such as Azure Sentinel, that support multiple instances, pivot now supports separate instance naming so that each Workspace has a separate instance of a given pivot query.

The naming of the Threat Intelligence pivot functions has been simplified considerably.

VirusTotal and RiskIQ relationships should now be available as pivot functions (you need the VT 3 and PassiveTotal packages installed respectively to enable this functionality).

Simplify imports in msticpy

The root module in msticpy now has several modules and classes that can be directly accessed from it (rather than having to import them)

python import msticpy as mp mp.init_notebook() qry_prov = mp.QueryProvider("MDE") ti = mp.TILookup()

Also a number of commonly-used classes are imported by default by init_notebook, notable all of the entity classes. This makes it easier to use pivot functions without any initialization or import steps.

entities ```python import msticpy as mp mp.init_notebook()

IpAddress.whois("123.45.6.78") ```

Consolidation of Pandas accessors

Pandas accessors are extensions to DataFrames allowing you to call custom functionality as a DataFrame method.

Almost all of the core MSTICPy functions previously available in various accessors (plus a few new ones) are accessible in:

df.mp - analysis and transform functions
df.mp_plot - visualization functions

```python3

df.mp.ioc_extract(...)
df.mp.to_graph(...)
df.mp.mask(...)

df.mp_plot.timeline(...)
df.mp_plot.timeline_values(...)
df.mp_plot.process_tree(...)
df.mp_plot.network(...)
df.mp_plot.folium_map(...)

```

MS Defender Queries available to MS Sentinel Query Provider

Since Sentinel now has the ability to import Microsoft data, we've made the Defender queries usable from the MS Sentinel provider.

Many of these queries are now available as Pivot functions.

ContiLeaks notebook added to MSTICPy Repo

We are privileged to host Thomas's awesome ContiLeaks notebook. Thanks @fr0gger

New Queries added

Several new Sentinel and MS Defender queries have been added.

See the new built-in query list

Documentation Additions and Updates

The documentation for V2.0 is now live and available at https://msticpy.readthedocs.io (Previous versions are still online and can be accessed through the ReadTheDocs interface).

New MSTICPy Quickstart Guide
Updated Installing guide
Updated Threat Intel Lookup documentation
Updated Time Series analysis documentation
New Plot Network Graph from DataFrame
Updated Plotting Folium maps
Updated Pivot functions
Updated Jupyter and Sentinel

The API documentation has been split into separate modules to make it easier to navigate. The API docs also now support "InterSphinx". This means that MSTICPy references to objects in other packages (e.g. Python standard library, pandas, Bokeh) have active links that will take you to the native documentation for that item.

Also, the sample notebooks for most of these features have been updated along the same lines. See MSTICPy Sample notebooks

Miscellaneous Improvements

The MS Sentinel provider now support a timeout parameter allowing you lengthen and shorten the default.
MSTICPy network requests use a custom User Agent header so that you can identify or track requests from MSTICPy/Notebooks.

Plus a lot more that I can't recall at the moment.

What's Changed - The gory detail of the PRs

Sync changes to main into v2 branch by @ianhelle in https://github.com/microsoft/msticpy/pull/330
Ianhelle/msticpy v2.0.0 merge updates 2022 03 14 by @ianhelle in https://github.com/microsoft/msticpy/pull/338
Ianhelle/implement isort 2022 02 15 by @ianhelle in https://github.com/microsoft/msticpy/pull/327
Ianhelle/implement isort branch post-fixes 2022 03 21 by @ianhelle in https://github.com/microsoft/msticpy/pull/346
Ianhelle/pivot dataprov selfload 2022 03 15 by @ianhelle in https://github.com/microsoft/msticpy/pull/343
Ianhelle/main mergeback 2022 04 05 by @ianhelle in https://github.com/microsoft/msticpy/pull/355
Merging changes from main for geoip.py, config editor and kusto_driver by @ianhelle in https://github.com/microsoft/msticpy/pull/359
Pebryan/2022 4 14 auth merge by @petebryan in https://github.com/microsoft/msticpy/pull/368
Fixed minor issues by @petebryan in https://github.com/microsoft/msticpy/pull/372
Ianhelle/v2 reorg directories 2 2022 04 12 by @ianhelle in https://github.com/microsoft/msticpy/pull/377
Ianhelle/mpconfigedit fix from main 2022 05 22 by @ianhelle in https://github.com/microsoft/msticpy/pull/396
Added pd accessor for time series functions. by @ianhelle in https://github.com/microsoft/msticpy/pull/381
Added new Sentinel Search Features - merge from main by @ianhelle in https://github.com/microsoft/msticpy/pull/380
Ianhelle/ti async lookup 2022 04 27 by @ianhelle in https://github.com/microsoft/msticpy/pull/383
Ianhelle/folium accessor 2022 04 30 by @ianhelle in https://github.com/microsoft/msticpy/pull/384
Updated tweet action to include more details by @petebryan in https://github.com/microsoft/msticpy/pull/406
Add Device Code fallback option for when interactive auth isn't avaliable. by @petebryan in https://github.com/microsoft/msticpy/pull/401
Adding OData Delegated Auth Support into 2.0 by @petebryan in https://github.com/microsoft/msticpy/pull/410
Removed plaintext token cache from MSAL auth and replaced it with fall back to in memory caching by @petebryan in https://github.com/microsoft/msticpy/pull/414
Ianhelle/kql nbinit fixes merge2.0 2022 05 18 by @ianhelle in https://github.com/microsoft/msticpy/pull/412
Ianhelle/geoip init fix 2022 05 27 by @ianhelle in https://github.com/microsoft/msticpy/pull/421
Ianhelle/geoip init fix 2022 05 27 by @ianhelle in https://github.com/microsoft/msticpy/pull/422
Ianhelle/geoip init fix 2022 05 27 by @ianhelle in https://github.com/microsoft/msticpy/pull/423
Ianhelle/read the docs fixes 2022 05 29 by @ianhelle in https://github.com/microsoft/msticpy/pull/424
Ianhelle/sentinel workspace lookup 2022 05 19 by @ianhelle in https://github.com/microsoft/msticpy/pull/419
Fix for listhuntingqueries function by @pensivepaddle in https://github.com/microsoft/msticpy/pull/417
Update calls to credential.modern.get_token by @FlorianBracq in https://github.com/microsoft/msticpy/pull/429
Adding ContiLeaks Analysis by @fr0gger in https://github.com/microsoft/msticpy/pull/428
Networkx graphs from dataframe by @ianhelle in https://github.com/microsoft/msticpy/pull/427
Ianhelle/msticpy init imports and Quickstart doc by @ianhelle in https://github.com/microsoft/msticpy/pull/435
Ianhelle/main updates to msticpy v2.0.0 2022 06 14 by @ianhelle in https://github.com/microsoft/msticpy/pull/444
[fix] Revert to Py 3.7 build with typing-extensions by @ianhelle in https://github.com/microsoft/msticpy/pull/448
[fix] if AuthKey or ApiID is None by @ianhelle in https://github.com/microsoft/msticpy/pull/449
Ianhelle/query pivot naming 2022 06 06 by @ianhelle in https://github.com/microsoft/msticpy/pull/437
Ianhelle/folium update docs 2022 05 29 by @ianhelle in https://github.com/microsoft/msticpy/pull/438
Ianhelle/timeline updates 2022 06 14 by @ianhelle in https://github.com/microsoft/msticpy/pull/441
Ianhelle/merge2.0 to main 2022 06 14 by @ianhelle in https://github.com/microsoft/msticpy/pull/443
[fix] Fixing renamed y/value_col parameter by @ianhelle in https://github.com/microsoft/msticpy/pull/452
[fix] Incorrect handling of kwargs in timeline_values.py by @ianhelle in https://github.com/microsoft/msticpy/pull/455
Ianhelle/ti and graph doc update 2022 06 15 by @ianhelle in https://github.com/microsoft/msticpy/pull/453
Ianhelle/mp user agent 2022 06 16 by @ianhelle in https://github.com/microsoft/msticpy/pull/454
Added features to support new notebooklet development. by @petebryan in https://github.com/microsoft/msticpy/pull/456

New Contributors

@pensivepaddle made their first contribution in https://github.com/microsoft/msticpy/pull/417
@fr0gger made their first contribution in https://github.com/microsoft/msticpy/pull/428

Full Changelog: https://github.com/microsoft/msticpy/compare/v1.8.2...v2.0.0.rc3

- Python
Published by ianhelle almost 4 years ago

msticpy - MSTICPy 2.0.0 pre-release 2

New Features

There are several new features in V 2.0.0 of MSTICPy. The major items include:

Folium map update - plot a map using multiple layers, custom icons, colors and tooltips from a single function call.
Time Series - calculate and display a Time Series anomalies plot from a single function call.
Threat Intelligence lookups - individual providers run asynchronously (simultaneously) making it many times faster to perform lookups across providers. Lookup progress is also displayed with a progress bar

Pre-release documentation for v2.0.0 is on ReadtheDocs Note: API documentation should be up-to-date but user-guides for new features are still TBD.

Folium map update

The Folium module in MSTICPy has always been a bit complex to use since it normally required that you convert IP addresses to MSTICPy IpAddress entities before adding them to the map. You can now plot maps with a single function call from a DataFrame containing IP addresses or location coordinates. You can group the data into folium layers, specify columns to populate popups and tooltips and to customize the icons and coloring.

folium_layers

plot_map

A new plot_map function (in the msticpy.vis.foliummap module) that lets you plot mapping points directly from a DataFrame. You can specify either an ip_column or coordinates columns (lat_column and long_column). In the former case, the geo location of the IP address is looked up using the MaxMind GeoLiteLookup data.

You can also control the icons used for each marker with the icon_column parameters. If you happen to have a column in your data that contains names of FontAwesome or GlyphIcons icons you can use that column directly. More typically you would combine the icon_column with the icon_map parameter. You can specify either a dictionary or a function. For a dictionary, the value of the row in icon_column is used as a key - the value is a dictionary of icon parameters passed to the Folium.Icon class. For a function, the icon_column value is passed to the function as a single parameter and the return value should be a dictionary of valid parameters for the Icon class. You can read the documentation for this function in the docs

plot_map pandas accessor

Plot maps from the comfort of your own DataFrame! Using the msticpy mp_plot accessor you can plot maps directly from a DataFrame containing IP or location information. The folium_map function has the same syntax as plot_map except that you omit the data parameter.

```python

df.mp_plot.folium_map(ip_column="ip", layer_column="CountryName")

```

pd_accessors

Layering, Tooltips and Clustering support

In plot_map and .mp_plot.folium_map you can specify a layer_column parameter. This will group the data by the values in that column and create an individually selectable/displayable layer in Folium. For performance and sanity reasons this should be a column with a relatively small number of discrete values.

Clustering of markers in the same layer is also implemented by default - this will collapse multiple closely located markers into a cluster that you can expand by clicking or zooming.

You can also populate tooltips and popups with values from one or more column names.

Classic interface

The original FoliumMap class is still there for more manual control. This has also been enhanced to support direct plotting from IP, coordiates or GeoHash in addition to the existing IpAddress and GeoLocation entities. It also supports layering and clustering.

Threat Intelligence Providers - Async support

When you have configured more than one TI provider, MSTICPy will execute requests to each of them asynchronously. This will bring big performance benefits when querying IoCs from multiple providers. Note: requests to individual providers are still executed synchronously since we want to avoid swamping provider services with multiple simultaneous requests.

We've also implemented progress bar tracking for TILookups, giving a visual indication of progress when querying multiple IoCs.

Combining the progress tracking with asynchronous operation means that not only is performing lookups for lots of observables faster but ou will also less likely to be left guessing whether or not your kernel has hung.

TI Providers are now also loaded on demand - i.e. only when you have a configuration entry in your msticpyconfig.yaml for that provider. This prevents loading of code (and possibly import errors) due to providers which you are not intending to use.

Finally, we've added functions to enable and disable providers after loading TILookup: - tilookup.enableprovider - tilookup.disableprovider - tilookup.setprovider_state

```python

from msticpy.context import TILookup
ti_lookup = TILookup()

iocs = ['162.244.80.235', '185.141.63.120', '82.118.21.1', '85.93.88.165']
ti_lookup.lookup_iocs(iocs, providers=["OTX", "RiskIQ"])

```

ti_providers_async

Time Series pandas accessor

Although the Time Series functionality was relatively simple to use, it previously required several disconnected steps to compute the time series, plot the data, extract the anomaly periods. Each of these needed a separate function import. Now you can do all of these from a DataFrame via pandas accessors. (currently there is a separate accessor df.mp_timeseries but we are still working on consolidating our pandas accessors so this may change before the final release.)

Because you typically still need these separate outputs, the accessor has multiple methods:

df.mp_timeseries.analyze - takes a time-summarized DataFrame and returns the results of a time-series decomposition
df.mp_timeseries.plot - takes a decomposed time-series and plots the anomalies
df.mp_timeseries.anomaly_periods - extracts anomaly periods as a list of time ranges
df.mp_timeseries.anomaly_periods - extracts anomaly periods as a list of KQL query clauses
df.mp_timeseries.apply_threshold - applies a new anomaly threshold score and returns the results.

See documentation

Analyze data to produce time series.

```python

df = qry_prov.get_networkbytes_per_hour(...)
ts_data = df.mp_timeseries.analyze()

```

Analyze and plot time series anomalies

```python

df = qry_prov.get_networkbytes_per_hour(...)
ts_data = df.mp_timeseries.analyze().mp_timeseries.plot()

```

Analyze and retrieve anomaly time ranges

```python

df = qry_prov.get_networkbytes_per_hour(...)
ts_data = df.mp_timeseries.analyze().mp_timeseries.anomaly_periods()

```

In next pre-release

Plot networks (graphs) directly from a DataFrame

One frequently-requested feature is the ability to easily plot networks from data. For example you may want to view the interactions between account names and IP addresses. This feature use Networkx to build the graph and Bokeh to plot the graph.

Note: The graph has the usual Bokeh interactivity - zoomin, panning, selecting, hover-over tooltips. It does not allow you to move individual nodes and interactively recalculate the layout. For the latter, you can use this functionality to build a networkx graph and plot using something like GraphViz or PyViz.

The network plot will give you two functions:

df.mp.to_graph to convert a DataFrame to a networkx graph
df.mp_plot.network create and plot the graph in a single step.

(There is also a separate function msticpy.vis.network_plot.plot_nx_graph that will just do the NX -> plot operation)

You can specify the columns to use as source and target. An edge is created between source and target when the two occur on in the same row (or more than one row). You can also specify columns to use as node and edge attributes.

To Do items

We intend to add the following before release:

allow you to specify the networkx layout algorithm to use (currently it uses the default spring_layout)
assign edge weight attribute based on number of rows contributing to an edge

MS Sentinel Workspaces API

Lets you query and resolve details for Sentinel workspaces. This is integrated into the MpConfigEdit and MpConfigFile utilities to let you lookup workspace details when you are editing your settings:

paste in a URL from the Sentinel Azure portal to populate workspace settings
or resolve full details from partial workspace such as the workspace ID.

Other important fixes

The API details for most of the MSTICPy functions were not being generated - this should now be fixed.

What's Changed (GitHub PR Summary)

Added pd accessor for time series functions. by @ianhelle in https://github.com/microsoft/msticpy/pull/381
Added new Sentinel Search Features - merge from main by @ianhelle in https://github.com/microsoft/msticpy/pull/380
Ianhelle/ti async lookup 2022 04 27 by @ianhelle in https://github.com/microsoft/msticpy/pull/383
Ianhelle/folium accessor 2022 04 30 by @ianhelle in https://github.com/microsoft/msticpy/pull/384
Updated tweet action to include more details by @petebryan in https://github.com/microsoft/msticpy/pull/406
Add Device Code fallback option for when interactive auth isn't available. by @petebryan in https://github.com/microsoft/msticpy/pull/401
Adding OData Delegated Auth Support into 2.0 by @petebryan in https://github.com/microsoft/msticpy/pull/410
Removed plaintext token cache from MSAL auth and replaced it with fall back to in memory caching by @petebryan in https://github.com/microsoft/msticpy/pull/414
Ianhelle/kql nbinit fixes merge2.0 2022 05 18 by @ianhelle in https://github.com/microsoft/msticpy/pull/412
Ianhelle/geoip init fix 2022 05 27 by @ianhelle in https://github.com/microsoft/msticpy/pull/421
Ianhelle/geoip init fix 2022 05 27 by @ianhelle in https://github.com/microsoft/msticpy/pull/422
Ianhelle/geoip init fix 2022 05 27 by @ianhelle in https://github.com/microsoft/msticpy/pull/423

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.0.0.rc1...v2.0.0.rc2

- Python
Published by ianhelle about 4 years ago

msticpy - Fixes for Linux auth, kql and nbinit initialization

Minor release fixing a few usability issues.

What's Changed

Adding full Delegated Auth support to all OData Drivers by @petebryan in https://github.com/microsoft/msticpy/pull/409 This allows MDE and Graph users to use User-delegated authentication rather than app ID/secrets
Fixes for usability bugs in kqldriver, nbinit, userconfig - added typing-extensions requirement by @ianhelle in https://github.com/microsoft/msticpy/pull/411
- Kql driver will revert to Kqlmagic-based device authentication if Azure Authentication fails
- Kql driver suppresses "missing PyGObject message" - a dependency that isn't required in this scenario
- init_notebook produced spurious error message about Virus Total libraries not being available even if they were not used.
- User config throws error if user has partial auto-load configuration in msticpy
Replace MSAL auth plaintext file cache with memory cache by @petebryan in https://github.com/microsoft/msticpy/pull/413
- removed ability to use plaintext token cache because of security concerns
Update API version for listalertrules by @FlorianBracq in https://github.com/microsoft/msticpy/pull/399
Updating Dockerfile source to mcr anaconda by @ianhelle in https://github.com/microsoft/msticpy/pull/397 Docker source switched to trusted anaconda source for supply chain security
Updated Tweet bot to include more context in the tweets by @petebryan in https://github.com/microsoft/msticpy/pull/403
Updated tweet action to include more detail in the tweets by @petebryan in https://github.com/microsoft/msticpy/pull/405
Adding Microsoft SECURITY.MD by @microsoft-github-policy-service in https://github.com/microsoft/msticpy/pull/407
Bump readthedocs-sphinx-ext from 2.1.5 to 2.1.6 by @dependabot in https://github.com/microsoft/msticpy/pull/400

Full Changelog: https://github.com/microsoft/msticpy/compare/v1.8.1...v1.8.2

- Python
Published by ianhelle about 4 years ago

msticpy - MSTICPy V2.0.0 Pre-Release 1

One of our main goals for V2.0.0 was to re-organize MSTICPy to be more logical and easier to use and maintain. Several years of organic growth had seen modules created in places that seemed like a good idea at the time but did not age well.

The discussion about the V2 structure can be found here #320.

This first pre-release of V2.0 has a small number of new features but is mostly about testing compatibility. We'll be adding features to future pre-releases in the next week or two.

*Due to the re-organization, many features are no longer in places where they used to be imported from! *

We have tried to maintain compatibility with old locations by adding "glue" modules. These allow import of many modules from their previous locations but will issue a Deprecation warning if loaded from here. The warning will contain the new location of the module - so you should update your code to point to this new location.

This table gives a quick overview of the V2.0 structure

| folder | description | |-----------|----------------------------------------------------------------------------------| | analysis | data analysis functions - timeseries, anomalies, clustering | | auth | authentication and secrets management | | common | common used utilities and definitions (e.g. exceptions) | | config | configuration and settings UI | | context | enrichment modules geoip, ip_utils, domaintools, tiproviders, vtlookup | | data | data acquisition/queries/storage/uploaders | | datamodel | entities, soc objects | | init | package loading and initialization - nbinit, pivot modules | | nbwidgets | nb widgets modules | | transform | simple data processing - decoding, reformatting, schema change, process tree | | vis | visualization modules including browsers |

Notable things that have moved:

most things from the sectools folder have migrated to context, transform or analysis
most things from the nbtools folder have migrated to:
- init - (not to be confused with init) - package initialization
- vis - visualization modules
pivot functionality has moved to init

If you are having trouble finding a module, we have added a simple search function

python import msticpy msticpy.search("riskiq")

Any matches will be returned in a table with links to the module documentation

Modules matching 'riskiq'

Module	Help
msticpy.context.tiproviders.riskiq	msticpy.context.tiproviders.riskiq

The Help link is to the current release version of the documentation, so will not map to the correct path on ReleaseTheDocs for the release documentation. You should be able to map this to the pre-release doc version by adding v2.0.0.rc1 into the URL.

https://msticpy.readthedocs.io/en/v2.0.0rc1/api/msticpy.sectools.tiproviders.html

Other Features/Changes

Pandas accessors have been consolidated into two main ones:

df.mp.func - contains all of the transformation functions like base64 decoding, ioc searching, etc.
df.mp_plot.func - contains all of the visualization accessors (timeline, process tree, etc.)

We've also done a lot of work to clean up warnings, move some remaining Python unittest tests to pytest format and random other things that you do when you're cleaning house. A fuller but not especially readable list of changes follows.

Please let us know if you hit any issues with the re-jigged structure or any other bugs. File them as issues or reach out to msticpy@microsoft.com

List of changes

Co-authored-by: Ian Hellen ianhelle@microsoft.com Co-authored-by: Pete Bryan pebryan@microsoft.com

857835d@Ianhelle/mpconfigedit fix from main 2022 05 22 (#396)* Fix for MpConfigEdit ValueError
Updating Dockerfile source to mcr anaconda
bd96d40@Ianhelle/v2 reorg directories 2 2022 04 12 (#377)* Merging changes from main for geoip.py, mpconfigedit, mpconfigfile, pkgconfig and kustodriver
Some fixes to Kusto common_imports
- now works with Kusto config entry without instance suffix
- can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query
- added explicit "database" key in query files - can be used instead of the more opaque "datafamily.database" encoding in the datafamilies key.
Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration
Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py.
Fixed bug in mpconfigedit.py and mpconfigfile.py where empty/new msticpyconfig.yaml didn't save any settings.
Reorganized logic for handling parameters and failing on invalid file path in config module.
Moved analysis.data to transform folder
Moved data.context to context
Moved auth and secrets modules to auth folder
Moved nbwidgets to new folder
Updated deprecation warning in glue modules to v2.2
Fix to URLs in README.md
Adding init folder - moved:
- nbinit.py, userconfig.py, pivot.py, azureml_tools.py to here
- also moved vtpivot.py and pivottiprovider.py to init/pivotinit
Renaming datamodel/pivots to datamodel/pivot
Moved azureblobstorage.py to data/storage folder
Refactored querycontainer to datatypes.py - to be separate types for queries (querycontainer.py) and pivots (pivotcontainer.py)
Moved browsers to vis folder
Updating API docs for moves
Moved all pivot functions to init folder.
Added functionality to pkg_config to delete and translate settings (for AzureSentinel->MSSentinel switch TBD)
Added automatic acquisition of globals() in nbinit.py
Fixed a couple of bugs in pivot_pipeline.py
Removed direct import of pivot into datamodel/pivot and added code to add them dynamically after init.pivot initiialization.
Added trap to timeline when supplied with no data.
Fixed incorrect escaping in regex in kql_driver.py
Notebook updates for errors and invalid links.
Added script to run all notebooks for testing
Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x). Other components use the default.
Updated typing rigor for pkgconfig::gethttp_timeout
Fixed bugs in test testpkgconfig.py and testcodeview.py
Fixed re-auth on query issue in KQL driver
Fixed kql_driver tests
Kusto provider
- now works with Kusto config entry without instance suffix
- can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query
- added explicit "database" key in query files - can be used instead of the more opaque "datafamily.database" encoding in the datafamiies key.
Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration
Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py.
Fixed bug in mpconfigedit.py and mpconfigfile.py where empty/new msticpyconfig.yaml didn't save any settings.
Reorganized logic for handling parameters and failing on invalid file path in config module.
e2e743a@Ianhelle/main mergeback 2022 04 05 (#355)* Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters.
Fixed query_time widget so that you can reset time range from parameter
Added additional unit test for QueryTime setter
Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line.
Changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName.
Updated TimeSpan class so it has more flexible constructor
Added accountid as a parameter for listaadsigninsby_account query
Fix for kqldriver - reconnecting for each query loses original kwargs (including mpazauth) setting, so reverts to defaults. This can cause errors if the defaults are different to user-specified parameters. There is also a problem in azureauth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings.
aa14784@moved listsentinelworkspaces to AzureData
f3e82d9@New MSAL delegated auth methods added and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from #352.
ba33c38@Ianhelle/pivot dataprov selfload 2022 03 15 (#343)* Query providers load pivots dynamically when created.
Renamed query paths and changed data_providers so that only env-specific queries are loaded.
Moving ensuredfdatetimes to common/data_utils.py to avoid circular imports
Consolidated data-related pandas accessors into single module.
Fixing circular dependency in iocextract
Fixing linting errors in dataproviders.py, azureresource.py, host.py, process.py, pivotdataqueries.py
Adding default timeout values to httpx calls.
Changing torexitnodes.py Tor provider to defer download of tor list until first lookup
Fixing test for trigger Tor node download before running test.
Fixing case where MpConfigEdit loads with no current msticpyconfig.yaml. (from main branch)
- Now loads with empty settings rather than throw exception.
- Added unit test case
Aligning splunk_uploader params with base class
243f196@Ianhelle/implement isort branch post-fixes 2022 03 21 (#346)* move query files
Stub files and restructure fixes
Merging in updates to Azure auth
Making tests more resilient for multiple environments
restructure cyberreason and splunk queries
Fixed incorrect vtlookup
Updating missed Conda version for respx
Updated cybereason folder names
Fixed Sentinel APIs
Updated test mocked data to match new Sentinel APIs
Renamed data.context_providers to context
Renamed data.common to data.core
Added docstrings to redirection files so that they link to right location in read-the-docs
Updated docs with new paths
Updated notebooks with new module paths
Changed RTD to generate an API page for each module.
Add text to deprecation warning that we'll remove in v2.0.0
Adding isort to pre-commit.yaml
Isorting all of the imports in msticpy, tests and tools
Renaming query folders
Removing some automatic imports from msticpy.init.py
Adding search function to find modules to utility.py
Fixing old paths in test_timeline.py
Adding triggers for release branches
Adding isort to requirements-dev and conda-reqs-dev.txt
Bandit FPs in anomaly sequence modules
Fixed failing clustering notebook
Errors in notebook and keyvault tests
Fixing test issues in MicrosoftDefender.ipynb and EventClustering.ipynb
Bug using wrong httpx code syntax in odata_driver.py
Removing auto-load of VTLookup in nbinit
Forcing notebook tests to use test msticpyconfig-test.yaml
Added missing init.py to tests/data/browsers
Removing vtlookup import from sectools init because of circular import error
Moving IPStack check for API key to first call (rather than init) to avoid error on load.
prospector config changed produces deprecation warning and non-zero exit code.
Fixing misconfigured prospector.yaml
Getting rid of warning from test_nbinit
Adding McCabe suppression to ip_utils.py
removing version restriction for prospector in Github actions python-package.yaml
Powershell viewer PR
Elastic driver skeleton and changes to allow driver-specific param substitution
Cleaning up some import redirections to point to new locations
Fixing circular import in vtfile_behavior
Adding placeholder class to allow imports to sort of work even if sub-modules fail to import
Refactored valueordefault in querysource to reduce complexity
Fixed import errors in elasticdriver.py and splunkdriver.py
Fixed import from old location in nbinit
Fixing warning in code_view.py
Fixed test failure in testcodeview
Changing testtimeline.py to use new mpplot accessor in place of deprecated one.
Fixing test failure in test_timeline.py
3f3140a@Sync changes to main into v2 branch (#330)* adding devcontainer files (#321)
Syncing recent IPython-related changes (to skip magic creation if not in IPython)
f7923e5@Minor quality of life improvements to AzureData & MicrosoftSentinel (#331)* Updated names in AzureData to match MicrosoftSentinel
Added connection checks to Sentinel features

PRs

Sync changes to main into v2 branch by @ianhelle in https://github.com/microsoft/msticpy/pull/330
Ianhelle/msticpy v2.0.0 merge updates 2022 03 14 by @ianhelle in https://github.com/microsoft/msticpy/pull/338
Ianhelle/implement isort 2022 02 15 by @ianhelle in https://github.com/microsoft/msticpy/pull/327
Ianhelle/implement isort branch post-fixes 2022 03 21 by @ianhelle in https://github.com/microsoft/msticpy/pull/346
Ianhelle/pivot dataprov selfload 2022 03 15 by @ianhelle in https://github.com/microsoft/msticpy/pull/343
Ianhelle/main mergeback 2022 04 05 by @ianhelle in https://github.com/microsoft/msticpy/pull/355
Merging changes from main for geoip.py, config editor and kusto_driver by @ianhelle in https://github.com/microsoft/msticpy/pull/359
Pebryan/2022 4 14 auth merge by @petebryan in https://github.com/microsoft/msticpy/pull/368
Fixed minor issues by @petebryan in https://github.com/microsoft/msticpy/pull/372
Ianhelle/v2 reorg directories 2 2022 04 12 by @ianhelle in https://github.com/microsoft/msticpy/pull/377
Ianhelle/mpconfigedit fix from main 2022 05 22 by @ianhelle in https://github.com/microsoft/msticpy/pull/396

Full Changelog: https://github.com/microsoft/msticpy/compare/v.1.5.2...v2.0.0.rc1

- Python
Published by ianhelle about 4 years ago

msticpy - Hotfix release for MpConfigEdit

What's Changed

MpConfigEdit throws exception when passed file name
MpConfigEdit throws error with invalid file path. by @ianhelle in https://github.com/microsoft/msticpy/pull/395

Full Changelog: https://github.com/microsoft/msticpy/compare/v1.8.0...v1.8.1

- Python
Published by ianhelle about 4 years ago

msticpy - Sentinel Search API, Azure Authentication, Settings management, HTTPX timeouts

Summary

There are some feature changes and fixes in this release: - MS Sentinel Search API support in the Sentinel package - allowing you to create, check status and delete automated search jobs. - Authentication updates to - support wider range of Azure authentication types (incl VSCode and Powershell) - specify tenantID at auth time (this was a specific issue for Azure Data explorer users) - lets you use MSAL token caching - Fixes to httpx timeouts (we recently switch from using requests to httpx and have changed the default timeout to be none (like requests). You can also set this in configuration and specify while calling a function that makes a network request, e.g. mde_prov.my_query(...params, timeout=30) - MpConfigEdit now behaves better when no existing msticpyconfig.yaml exists - making it easier to create a config file from scratch.

What's Changed

Ianhelle/mp config edit load fix 2022 03 28 by @ianhelle in https://github.com/microsoft/msticpy/pull/352 Also fixes to multiple widgets for papermill/automation. Can now set QueryTime timespan programmatically. Fix to Sentinel data provider to prevent re-auth for every query.
Pebryan/2022 3 29 auth updates by @petebryan in https://github.com/microsoft/msticpy/pull/351
- Re-implemented using DefaultCredential
- Added support for MSAL token cache
Bump sphinx from 4.4.0 to 4.5.0 by @dependabot in https://github.com/microsoft/msticpy/pull/350
Fixes for GeoLiteLookup and MpConfigEdit by @ianhelle in https://github.com/microsoft/msticpy/pull/356
- GeoIPLite no longer tries to update DB during initialization - only on first query
- Fixes to MpConfigEdit and MpConfig file for msticpyconfig path handling.
Some fixes to Kusto common_imports by @ianhelle in https://github.com/microsoft/msticpy/pull/358
- Kusto queries now support "database" element in query file. This controls DB usage rather than having to encode inside the datafamily. Read more here https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProv-Kusto.html
Changing the pattern for httpx timeout to default to Timeout(None). by @ianhelle in https://github.com/microsoft/msticpy/pull/378
- config based setting for http timeouts
- can be overridden in calls in several providers - OData (MSGraph, MDE) and HTTP-based TI providers.
Add Workflow to Tweet by @petebryan in https://github.com/microsoft/msticpy/pull/369
- testing to automatically tweet status on PR completion
Fixed minor issues by @petebryan in https://github.com/microsoft/msticpy/pull/371
- some issues with incident exploration having incomplete information
Fixing bug in localdatadriver.py if CSV with no TimeGenerated field by @ianhelle in https://github.com/microsoft/msticpy/pull/374
- also added new Sentinel query listlogonattemptsbyip
Ianhelle/nb fixes 2022 04 20 by @ianhelle in https://github.com/microsoft/msticpy/pull/379
Added new Sentinel Search Features: by @petebryan in https://github.com/microsoft/msticpy/pull/376
- Sentinel Search API - can create, delete and check status of an automated search
- Also added feature to treat cases where you only have one Sentinel workspace configured - this will be treated as the default even if it is not marked as the default.

Full Changelog: https://github.com/microsoft/msticpy/compare/v1.7.5...v1.8.0

- Python
Published by ianhelle about 4 years ago

msticpy - Splunk async queries, powershell deobfuscator and viewer.

Introduction

An intermediate release while we prepare the ground for v2.0.0 of MSTICPy. The release includes some small features a several fixes

What's Changed

Added Splunk async provider and unit_tests by @d3vzer0 in https://github.com/microsoft/msticpy/pull/337 The default is now to execute splunk queries asynchronously - the previous default was to run all queries as one-shot queries so limiting the size of results sets that could be returned. To use the previous behaviour add the oneshot=True parameter to the query function.
Big thanks to @d3vzer0 (Joey Dreyer) for this (in addition to previous Splunk queries!)
Powershell simple de-obfuscator and code viewer. by @ianhelle in https://github.com/microsoft/msticpy/pull/335 python obfus_str = """ IF($PSVerSIONTAble.PSVErSion.MaJoR -gE 3){$43de2=[REF].AsSEmbly.GetTypE(\'System.Management.Automation.Utils\')."GEtFIe`LD"(\'cachedGroupPolicySettings\',\'N\'+\'onPublic,Static\');IF($43DE2) ... """ from msticpy.analysis.code_cleanup import format_powershell from msticpy.vis.code_view import display_html display_html( format_powershell(obfus_str), language="powershell" ) output powershell if($psversiontable.psversion.major -ge 3) { $43de2=[ref].assembly.gettype('system.management.automation.utils')."getfield"('cachedgrouppolicysettings','nonpublic,static') if($43de2) { $712db=$43de2.getvalue($null) if($712db['scriptblocklogging'])
Fixed incorrect use of httpx.codes enum in odata_driver by @ianhelle in https://github.com/microsoft/msticpy/pull/345
Added refresh and delete functions for keyring cached secrets by @ianhelle in https://github.com/microsoft/msticpy/pull/336
Bump readthedocs-sphinx-ext from 2.1.4 to 2.1.5 by @dependabot in https://github.com/microsoft/msticpy/pull/339

Other Fixes and minor changes

3af4379@ Bump readthedocs-sphinx-ext from 2.1.4 to 2.1.5 (#339)Bumps [readthedocs-sphinx-ext]

c0fc7fa@ * Fixed incorrect use of httpx.codes enum in odatadriver * Added httpx timeouts (30sec connect, 10sec other) to cybereasondriver and http_base and other users of httpx * Splitting keyring into its own module so that we can load without Keyring as a dependency - avoiding load failure on linux systems without keyring support * Prevent VTLookupV3 from trying to load automatically and throwing exception * Moved IPStack check for API key out of constructor so avoid error if loaded with no configuration. * Change Msticpy exceptions to output plaintext if not running in a notebook (e.g. IPython) * Added -utf16 argument to %b64 magic to allow correct decoding of Windows strings

- Python
Published by ianhelle about 4 years ago

msticpy - Hackmonth updates: CyberReason, Insights, Splunk, AWS, Kusto

Highlights

We have two new providers - a Threat intel provider for IntSights and a data/query provider for Cybereason contributed by Florian Bracq - AXA (@FlorianBracq). The Cybereason docs are here.

A range of new Splunk queries for our Splunk data provider for Alert and Authentication data contributed by Joey Dreijer (@d3vzer0)

Two cool sample notebooks: - Azure data explorer/Kusto notebook from Liam Kirton @liamkirton - AWS S3 HoneyBucket Log analysis notebook by Ashwin Patil @ashwin-patil

Complete replacement of requests with the async-capable httpx library by GrantV @grantv9

New Sentinel Security alert entity that automatically extracts related entities from a MS Sentinel alert.

Overhaul of some of our documentation - including automated build/collection of API docs and data provider query list (so these should now be up-to-date) and re-organization/re-write of the Data provider docs.

Reorganization of setup and tool settings using setup.cfg and pyproject.toml.

We also added a workaround (the IPython version dependency) to allow msticpy to be installed on Python 3.6 (even though we don't officially support it).

What's Changed

Add IntSights support by @FlorianBracq in https://github.com/microsoft/msticpy/pull/276
Setup.cfg and Pyproject.toml by @ianhelle in https://github.com/microsoft/msticpy/pull/287
Ianhelle/nbwidgets refactor 2022 01 25 by @ianhelle in https://github.com/microsoft/msticpy/pull/299
Add tenant_id parameter back by @liamkirton in https://github.com/microsoft/msticpy/pull/295
(fix): typo in README by @zscholl in https://github.com/microsoft/msticpy/pull/294
Add alert-oriented queries for Splunk using datamodels by @d3vzer0 in https://github.com/microsoft/msticpy/pull/297
Add authentication-oriented queries for Splunk using datamodels by @d3vzer0 in https://github.com/microsoft/msticpy/pull/298
Fix QueryTime unit change by @kubajir in https://github.com/microsoft/msticpy/pull/303
Update regular expression to extract parameters from source by @FlorianBracq in https://github.com/microsoft/msticpy/pull/291
Ianhelle/data docs rewrite 2022 01 27 by @ianhelle in https://github.com/microsoft/msticpy/pull/304
SentinelAlert entity creation by @petebryan in https://github.com/microsoft/msticpy/pull/293
Replaced requests with httpx 0.18 by @grantv9 in https://github.com/microsoft/msticpy/pull/285
Updating httpx requirements to work with out requirements by @petebryan in https://github.com/microsoft/msticpy/pull/312
Change to Querylist doc and py script to avoid phantom changes by @ianhelle in https://github.com/microsoft/msticpy/pull/310
Cybereason by @FlorianBracq in https://github.com/microsoft/msticpy/pull/306
AWS S3 HoneyBucket Log analysis notebook by @ashwin-patil in https://github.com/microsoft/msticpy/pull/311
Kusto Driver Example Notebooks by @liamkirton in https://github.com/microsoft/msticpy/pull/308
Bump sphinx from 4.3.2 to 4.4.0 by @dependabot in https://github.com/microsoft/msticpy/pull/283
doc updates by @ashwin-patil in https://github.com/microsoft/msticpy/pull/316
Adding dev container files for VSCode by @ashwin-patil in https://github.com/microsoft/msticpy/pull/321
Bump respx from 0.17.1 to 0.19.2 by @dependabot in https://github.com/microsoft/msticpy/pull/314
Updated Cybereason docs to fit pattern by @petebryan in https://github.com/microsoft/msticpy/pull/324
Ianhelle/1.6.1 hotfixes 2022 01 27 for 1.6.2 by @ianhelle in https://github.com/microsoft/msticpy/pull/317
Changing magics creation so that they don't get created if not in ipython by @ianhelle in https://github.com/microsoft/msticpy/pull/332

New Contributors

@FlorianBracq made their first contribution in https://github.com/microsoft/msticpy/pull/276
@zscholl made their first contribution in https://github.com/microsoft/msticpy/pull/294
@d3vzer0 made their first contribution in https://github.com/microsoft/msticpy/pull/297
@kubajir made their first contribution in https://github.com/microsoft/msticpy/pull/303
@grantv9 made their first contribution in https://github.com/microsoft/msticpy/pull/285

Full Changelog: https://github.com/microsoft/msticpy/compare/v1.6.1...v1.7.0

- Python
Published by ianhelle over 4 years ago

msticpy - v1.6.1 - Sentinel APIs, Folium, VirusTotal, ProcessTree, and more

In case you're wondering - we never released 1.6.0 to PyPI so this is the official 1.6.0 release.

Summary

Adding Clustering, SubGrouping, Geohash decoding and Layering to FoliumMap class (#227)

The FoliumMap class now supports layering and subgrouping. This was contributed by @tj-senserva (from Senserva)

Adding New Features to Sentinel APIs (#280)

Sentinel APIs has had signficant rework and expansion to cover new public APIs. These include Analytics and Watchlists as well as Incidents and Bookmarks from earlier releases. Each API now includes modification operations such as Create, Delete, Update (not all types support all types of modification).

Note: the module and class names drop the "Azure" prefix following the renaming of Azure Sentinel to Microsoft Sentinel

VirusTotal FileBehavior, FileBrowser and Pivot functions (#260)

The VirusTotal V3 module has been expanded to add additional operations and data visualization: - VT FileBehavior class lets you download and view file detonation data. - VT ObjectBrowser lets you download and view basic attributes of a file object - Pivot functions - these expose the VT3 Relationship APIs allowing you to pivot between related IPs, URLs, Files and Domains.

Added instance handling to Dataproviders (#218)

Several providers now support instances - e.g. Splunk, Sumologic can support multiple configuration entries in msticpyconfig.yaml with the use of an instance suffix, e.g. Spunk-Env1, Splunk-Evn2. This change also makes it possible to combine these instances into a single aggregate data provider and run the same query across the combined instances. This also works for multiple Microsoft Sentinel workspaces.

Process tree visualization updates to make schema-agnostic (#256)

The process tree visualization code previous had several dependencies on Sentinel or MDE schema specifics. The code should now accept process event data sets from many sources.

Adding Sysmon ProcSchema (#267) - A schema mapping for Sysmon data was contribute by @nbareil

Build pipeline fixes for external forks (#270)

Previously builds triggered from external forks would fail for a variety of reasons (no access to pipeline secrets, MS-Internal-only build actions). This is now fixed so that external contributors should expect a clean build.

Added default caching option for mordor driver (#257)

The OTRF Security Datasets driver and browser now cache security data set and MITRE ATT&CK data for quicker startup. By default, the cache files are stored in $HOME/.msticpy/mordor or the location specified in the Dataproviders/Mordor section of the msticpyconfig.

What's Changed

Ianhelle/proc tree fixes 2022 01 03 by @ianhelle in https://github.com/microsoft/msticpy/pull/256
Bump sphinx from 4.2.0 to 4.3.2 by @dependabot in https://github.com/microsoft/msticpy/pull/241
Adding Clustering, SubGrouping, Geohash decoding and Layering to FoliumMap class by @tj-senserva in https://github.com/microsoft/msticpy/pull/227
20220105 hackathon blob storage conn string by @lucky-luk3 in https://github.com/microsoft/msticpy/pull/262
Added instance handling to Dataproviders by @ianhelle in https://github.com/microsoft/msticpy/pull/218
Added default caching option for mordor driver by @ianhelle in https://github.com/microsoft/msticpy/pull/257
Ianhelle/virustotalv3 additions 2021 10 22 by @ianhelle in https://github.com/microsoft/msticpy/pull/260
Initial attempt at masking failing tests for external forks. by @ianhelle in https://github.com/microsoft/msticpy/pull/270
Adding Sysmon Process Create's ProcSchema by @nbareil in https://github.com/microsoft/msticpy/pull/267
Removing validate configuration for external forks by @ianhelle in https://github.com/microsoft/msticpy/pull/273
Update AzureData to use correct Azure subscription management API by @liamkirton in https://github.com/microsoft/msticpy/pull/269
Added details to README on upgrading msticpy by @danielc-evans in https://github.com/microsoft/msticpy/pull/274
Updated IPython elements and requirements by @petebryan in https://github.com/microsoft/msticpy/pull/278
Fixes and improvements to KqlDriver and KustoDriver Azure auth. by @liamkirton in https://github.com/microsoft/msticpy/pull/277
Miscellaneous fixes for VT, Auth and other items by @ianhelle in https://github.com/microsoft/msticpy/pull/281
Adding New Features to Sentinel APIs by @petebryan in https://github.com/microsoft/msticpy/pull/280
Correcting error introduced in nbinit refactoring by @ianhelle in https://github.com/microsoft/msticpy/pull/286

New Contributors

@dependabot made their first contribution in https://github.com/microsoft/msticpy/pull/241
@tj-senserva made their first contribution in https://github.com/microsoft/msticpy/pull/227
@lucky-luk3 made their first contribution in https://github.com/microsoft/msticpy/pull/262
@nbareil made their first contribution in https://github.com/microsoft/msticpy/pull/267
@liamkirton made their first contribution in https://github.com/microsoft/msticpy/pull/269
@danielc-evans made their first contribution in https://github.com/microsoft/msticpy/pull/274

Additional feature details

7fa1598@ - Add ability to filter queries by substring in list_queries in dataproviders.py - Adding coordinates property to GeoLocation entity geolocation.py - Simplify verbose option and refactoring complex method in initnotebook c0a32a0@Merge pull request #277 from liamkirton/liamkirton/kql-kusto-driver-auth - Fixes and improvements to KqlDriver and KustoDriver Azure auth. ae6bcd7@Updated IPython elements and requirements (#278)* Updated IPython elements and requirements 79910e2@Merge pull request #274 from danielc-evans/updated-readmeAdded details to README on upgrading msticpy 7038e28@Changed type hints in enumparse in utility.py - Added autoload of Pivot and vtpivot (if VT config detected) - Moved some common methods to IPyDisplayMixin class in nbwidgets.py - Added extra parameter for VT Private API definition to mpconfigdefaults.yaml. Also changed defaults for some providers to Primary=False - Updates to testmpconfigcontrols.py to deal with new VT parameter - Add fixed vtobjectbrowser.py + unit test in test_vtlookupv3.py 8308849@Added the ability to connect to a blob storage using the connection string. Container name parameter changed to optional.

Additional fix details

7fa1598@Miscellaneous fixes for VT, Auth and other items (#281)* Updates to Timeseries analysis and plotting modules - Avoiding exception when cmdline is NaN or non-string in processtree.py - Removing "Authenticating to Azure" output since it fires on every query. - Removing credential caching in azureauth_core.py

f730ba2@Moved geoip GeoLiteLookup to check for and update DB before first query (rather than in init) Added full copy of GeoLite2-City.mmdb to test data Linting fixes in mordordriver.py and test-testmordor_driver.py Cherry picked commit for python-package and azure-pipelines yamls

569eb75@Forcing install of pandas>=1.30 in pytest block 857b08e@Saved mdeprocpub.pkl as pickle protocol 4 to work in Python 3.6 70b5c55@Adding updated notice file. 4599d6e@Fixed use of hard-code global endpoint in azureblobstorage.py

cad8256@Added create incident and create rule features 70b5c55@Adding upgrade to pandas to so that it uses latest version and avoids pickle compatability issue with test data pickled with pandas > 1.2.5Also added higher version constraint to requirements-dev.txt and conda-reqs-dev.txt Adding updated notice file.

4599d6e@Added instance handling to Dataproviders (#218) - Fixed use of hard-code global endpoint in azureblobstorage.py - Adding better handling of response data in kqldriver.py - Removing default start and end parameters from queries. We should rely on the builtin querytimes control - Adding issue tracker URL and additional classifiers to setup.py - Updating importanalyzer.py to include resolved and unresolved paths for determining whether a package belongs to the std library. (these paths may be different if some of them are links) 0b9c81a@Bump sphinx from 4.2.0 to 4.3.2 (#241)

Full Changelog: https://github.com/microsoft/msticpy/compare/v.1.5.2...v1.6.1

- Python
Published by ianhelle over 4 years ago

msticpy - Config correction for Splunk driver

Hotfix release to correct incorrect capitalization reading values for Splunk settings

- Python
Published by ianhelle over 4 years ago

msticpy - MDE ProcTree fixes, Azure ML notebooks SSO

Summary

This release includes two minor changes: 1. Fixed schema for Microsoft Defender Device Process events - this should allow it to work with the Process Tree visualization 2. Single-sign-on in Azure ML notebooks using MSI

Fixes and New Features

MDE Process Tree

The schema originally deployed for generating process tree data from Microsoft Defender for Endpoint data was incorrect - we were using an internal schema that did not match the data retrieved via the Defender APIs. This has now been fixed along with some related items: - DateTime fields returned in the Defender data are automatically converted to pandas Timestamps (previously they were left as strings) - The process tree and other visualizations will automatically convert required timestamp columns to datetime format - this is useful where data is retrieved from an API or file where the required datetime columns are stored as date strings

Single Sign-On for notebooks in Azure ML (in Microsoft Sentinel)

When you sign on an Azure ML workspace, Managed Identity (MSI) credentials are automatically enabled on the Azure Computes used to run the notebooks. The MSI credentials use the user principal of the user that has logged on (not the machine identity). This allows processes on the Computes to use these credentials to authorize to other Azure resources. The Log Analytics team made some changes to allow the Managed Identity credentials from Azure ML to authorize to Log Analytics/Microsoft Sentinel data API.

With this release, connecting to Microsoft Sentinel will automatically try to use the MSI credentials. Existing authentication flow options such as Azure CLI and Device Code authentication are still available.

To use MSI-based SSO: python qry_prov = QueryProvider("AzureSentinel") # we will change this to MicrosoftSentinel soon! qry_prov.connect(WorkspaceConfig())` To force specific authentication flavors, you can set your preferences explicitly in the msticpconfig.yaml. You can also do this at the command line.

To specify an explicit set of authentication types to use (they are tried in the order specified): python qry_prov.connect(WorkspaceConfig(), mp_az_auth=["cli", "interactive"])

To skip integrated Azure authentication altogether and use interactive Device Code authentication: ```python qryprov.connect(WorkspaceConfig(), mpaz_auth=False)

or

qryprov.connect(WorkspaceConfig(), mpaz_auth=["interactive"]) ```

Detailed changes

4fdf286@Ianhelle/mde proctree fixes 2021 12 16 (#239)* Changes:

Removed checking of DataFamily in dataqueryreader.py
Changed default behavior of kql_driver.py to use Azure authentication
Change securitybase.py to use entities.OSFamily rather than querydefns.DataFamily
Added ensuredfdatetimes function to auto-convert specified columns from string to datetime
Added unit test testquerydefns.py
Added ensuredfdatetime to timeline.py
Added ensuredfdatetime to timeline_duration.py
Added ensuredfdatetime to proctreebuild_mde.py
Added ensuredfdatetime to proctreebuild_winlx.py
Changed mpconfig_defaults.yaml to include "msi"
Switching "Microsoft Sentinel" to "Azure Sentinel" in UI/messages.
Updated black version in .pre-commit-config.yaml
Using returned schema in mdatp_driver.py to auto-format datetimes to pandas timestamp
Returning full response from querywithresults in odata_driver.py
Some fixes in convertmdeschematointernal - field naming and using tz-aware Unknown time value
Moved the proc tree schema code from proctreebuilder.py to proctreeschema.py
Add mocked azconnect to testkqldriver.py tests - since the change to defaulting to AZ auth in the kqldriver means that tests fail.
Added test case for MDE public data - mdeprocpub.pkl
Adding tooltips to settings editor buttons
also changing button text for simple setting editor to "Update" (from "Save")
changing order of checks in base64unpack to let you specify UTF-16 decoding
updating version to 1.5.1

- Python
Published by ianhelle over 4 years ago

msticpy - RiskIQ, Sentinel Incident Explorer, Kusto and MS Defender

Release Summary

There are several cool new features in this release: - Microsoft Sentinel Incident exploration - RiskIQ Threat Intellligence provider and pivot functions - Microsoft Defender 365 API/Data provider support - Preview Azure Data Explorer/Kusto data provider support - Microsoft graph provider now supports sovereign clouds - ProcessTree visualization support for Microsoft Defender data in Microsoft Sentinel

Microsoft Sentinel Incident Exploration

0037dd6@Merge pull request #211 from microsoft/pebryan/2021-8-18_IncidentTriage

Provides new features to support Incident Triage scenarios (as included in the new Microsoft Sentinel Incident Triage notebook https://github.com/Azure/Azure-Sentinel-Notebooks/pull/120).

Entity Graph Visualization

This new visualization allows for the graphing of an investigation including any alerts, incidents or entities associated with that investigation. The graph can be created, updated and visualized either as a graph or with a graph and a timeline of the graph event times.

python from msticpy.vis.entity_graph_tools import EntityGraph graph = EntityGraph(incident) graph.plot()

A sample entity graph

The Entity Graph can also be created from a dataframe of events, either by passing in the dataframe or as a Pandas accessor.

python df.mp_plot.incident_graph(timeline=True)

A sample entity graph with timeline

Microsoft Sentinel Incident API support

RiskIQ PassiveTotal TI provider and Pivot provider

9353d9f@RiskIQ threat intel provider & pivots (#175)

Delivers a new "RiskIQ" threat intel provider, enabling access to RiskIQ Summary Card and Reputation datasets for IPs and hostnames with the same syntax and response format used with other TI Providers. python ti_lookup = TILookup() lookup_result = ti_lookup.lookup_ioc(observable="137.184.153.130", providers="RiskIQ") ti_lookup.result_to_df(lookup_result).T

Introduces a new mechanism for registering MSTICPy Pivot Functions provided by threat intel providers, then uses that code to register capabilities for nearly all RiskIQ API endpoints on hostnames, FQDN and IPAddress entities. These pivot functions enable easy integration of RiskIQ-provided intelligence in existing pivot pipelines, and makes it easy to build new pipelines leveraging unique RiskIQ datasets. python ( Dns.RiskIQ.resolutions("jquery.su") .query("recordtype=='A'") .mp_pivot.run(Dns.RiskIQ.reputation, column='resolve', join='left') .mp_pivot.run(IpAddress.RiskIQ.resolutions, column='resolve', join='left') )

Preview support for Kusto (Azure Data Explorer)

98864aa@Ianhelle/kusto and md365 support 2021 11 01 (#225)

This release contains a preview (i.e. not fully baked) of support for Kusto cluster queryingand Microsoft Defender 365 data providers. The Kusto/Azure Data Explorer support is based on the existing Microsoft Sentinel provider. python kusto_provider = QueryProvider("Kusto") kusto_provider.connect( "azure_data-Explorer://code;cluster='my_cluster';database='my_db';alias='my_alias'" ) Some more documentation to follow about configuration in msticpyconfig and how to build template queries for Kusto/Azure Data explorer.

Support for Microsoft 365 Defender data providers

This is slight expansion of our current support for MS Defender for endpoint. Using the new "M365D" provider name will cause the provider to use the new combined Microsoft Defender APIs, giving you consolidated access to data from multiple Defender services.

To create a provider for Microsoft 365 Defender: python m365d_provider = QueryProvider("M365D") m365d_provider.connect()

All template queries for that were available for MDE are also loaded for M365D.

Using Microsoft Defender for Endpoint is still supported. To use only the Microsoft Defender for Endpoint APIs use "MDE".

You can specify your connection parameters in msticpconfig.yaml yaml DataProviders: # ... MicrosoftDefender: Args: ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa ClientSecret: KeyVault: TenantId: 8360dd21-0294-4240-9128-89611f415c53 This configuration expects to find the client secret in your configured Key Vault with the name "DataProviders-MicrosoftDefender-Args-ClientSecret" - the msticpy config editor will create this for you. You can store the secret in your config file but, obviously, we would not recommend this.

You can also specify configuration/secret values in the call to m365d_provider.connect(). For more information see the Defender provider MSTICPy documentation

Other improvements

The Graph provider now supports sovereign clouds as well as the Azure global cloud. You can configure settings for MS Defender and MS Graph via the MSTICPy settings editor. If you have configured settings for these you can now connect with a simplified syntax, with no parameters required. python graph_prov = QueryProvider("SecurityGraph") graph_prov .connect()

The schema for the configuration is the same format as the Defender format described above.

yaml DataProviders: # ... MicrosoftGraph: Args: ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa ClientSecret: KeyVault: TenantId: 8360dd21-0294-4240-9128-89611f415c53

Process Tree visualization supports MS Sentinel schema for MDE process data

The schema of the DeviceProcessEvents table in Microsoft Sentinel is different from the data queried directly from the MS Defender advance hunting API. This update allows you to query MS Defender process data from Azure Sentinel and display in the Process Tree visualization.

New CI Pipelines using GitHub Actions and Azure Pipelines

The old Azure DevOps pipeline has been decommissioned and replaced by two new ones - GitHub actions and an improved version of the Azure DevOps pipeline.

The main advantage of the GitHub actions pipeline is that the results of tests and code checks are visible to external contributors.

However, there are some component governance and security checks that we are required to run by Microsoft but which are not currently available in GitHub actions. We've opted to run both in parallel.

The new Azure DevOps pipeline is more sophisticated and comprehensive than the previous one. It allows us to run builds and tests on Windows, Linux and Mac platforms and to cover Python versions 3.6 and 3.8. (Note: we will likely switch to 3.8 and 3.10 in the near future)

Other changes and fixes

98864aa@Ianhelle/kusto and md365 support 2021 11 01 (#225) - Fixing issue in odatadriver.py where apiroot would keep getting suffix appended if you connected multiple times. - Adding mutli-cloud support for MS Graph API in securitygraphdriver.py - Fixing syntax errors in graphalerts.yaml - Updating tests for testdrivers and testkqldriver - Fixed an issue in kqldriver with Kusto returning a different status code format 1c2c1be@Disabling IPstack tests because of restricted API allowance (#224)* - Disabling IPstack tests because of restricted API allowance - Fixing incorrect assignment in testgeoip.py - 56fcb81@Updating readme badge to reflect new build pipeline - cdff1ff@Update sphinx requirements in azure-pipelines.yml for Azure Pipelines - ce8cbe1@Update README.md to include downloads - 0cb20fd@Updated way of getting instance and domain name for azuremltools.py (#217) - Updated checking for version of msticpy using setuptools/pkgresources in azuremltools.py - Added warning if running with stale import of newly updated msticpy. - 8043207@Ianhelle/timeline args check 2020 09 21 (#219)* Adding some error checking of column names to timelinevalues and timelineduration - a7498a3@Ianhelle/azure pipelines yaml 2021 09 09 (#201)* - Pylint and mypy fixes in: - anomaloussequence/model.py - anomaloussequence/utils/cmdsparamsvalues.py - common/pkgconfig.py - common/wsconfig.py - config/compedit.py - config/mpconfigfile.py - data/dataproviders.py - data/dataqueryreader.py - data/querystore.py - data/sqltokql.py - datamodel/entities/host.py - datamodel/entities/ipaddress.py - datamodel/pivotregisterreader.py - nbtools/morphcharts.py - nbtools/nbdisplay.py - nbtools/securityalert.py - nbtools/securitybase.py - sectools/auditdextract.py - sectools/base64unpack.py - sectools/cmdline.cmdline.py - Adding pylint --disable=duplicate-code - Updating pylint requirements in requirements-dev.txt and .pre-commit-config.yaml - Add [withpyroma] to prospector install - Duplicating @noqa MC0001 suppressions since prospector/McCabe now seem to ignore decorators an locate the offending line as the "def" line rather than the decorator line. - Fixing MyPy-discovered error in nbwidgets.AlertSelector 9f122ee@refactored code in testgeoip 907e486@Remove un-needed code in testgeoip 8ae01f7@Updated testgeoip to fix failure case e5aadac@Added missing parameters to listallsigninsgeo

- Python
Published by ianhelle over 4 years ago

msticpy - Pre-Release: RiskIQ, Sentinel Incident Explorer, Kusto and MS Defender

Release Summary

There are several cool new features in this release: - Microsoft Sentinel Incident exploration - RiskIQ Threat Intellligence provider and pivot functions - Microsoft Defender 365 API/Data provider support - Preview Azure Data Explorer/Kusto data provider support - Microsoft graph provider now supports sovereign clouds - ProcessTree visualization support for Microsoft Defender data in Microsoft Sentinel

Microsoft Sentinel Incident Exploration

0037dd6@Merge pull request #211 from microsoft/pebryan/2021-8-18_IncidentTriage

Provides new features to support Incident Triage scenarios (as included in the new Microsoft Sentinel Incident Triage notebook https://github.com/Azure/Azure-Sentinel-Notebooks/pull/120).

Entity Graph Visualization

This new visualization allows for the graphing of an investigation including any alerts, incidents or entities associated with that investigation. The graph can be created, updated and visualized either as a graph or with a graph and a timeline of the graph event times.

python from msticpy.vis.entity_graph_tools import EntityGraph graph = EntityGraph(incident) graph.plot()

A sample entity graph

The Entity Graph can also be created from a dataframe of events, either by passing in the dataframe or as a Pandas accessor.

python df.mp_plot.incident_graph(timeline=True)

A sample entity graph with timeline

Microsoft Sentinel Incident API support

RiskIQ PassiveTotal TI provider and Pivot provider

9353d9f@RiskIQ threat intel provider & pivots (#175)

Delivers a new "RiskIQ" threat intel provider, enabling access to RiskIQ Summary Card and Reputation datasets for IPs and hostnames with the same syntax and response format used with other TI Providers. python ti_lookup = TILookup() lookup_result = ti_lookup.lookup_ioc(observable="137.184.153.130", providers="RiskIQ") ti_lookup.result_to_df(lookup_result).T

Introduces a new mechanism for registering MSTICPy Pivot Functions provided by threat intel providers, then uses that code to register capabilities for nearly all RiskIQ API endpoints on hostnames, FQDN and IPAddress entities. These pivot functions enable easy integration of RiskIQ-provided intelligence in existing pivot pipelines, and makes it easy to build new pipelines leveraging unique RiskIQ datasets. python ( Dns.RiskIQ.resolutions("jquery.su") .query("recordtype=='A'") .mp_pivot.run(Dns.RiskIQ.reputation, column='resolve', join='left') .mp_pivot.run(IpAddress.RiskIQ.resolutions, column='resolve', join='left') )

Preview support for Kusto (Azure Data Explorer)

98864aa@Ianhelle/kusto and md365 support 2021 11 01 (#225)

This release contains a preview (i.e. not fully baked) of support for Kusto cluster queryingand Microsoft Defender 365 data providers. The Kusto/Azure Data Explorer support is based on the existing Microsoft Sentinel provider. python kusto_provider = QueryProvider("Kusto") kusto_provider.connect( "azure_data-Explorer://code;cluster='my_cluster';database='my_db';alias='my_alias'" ) Some more documentation to follow about configuration in msticpyconfig and how to build template queries for Kusto/Azure Data explorer.

Support for Microsoft 365 Defender data providers

This is slight expansion of our current support for MS Defender for endpoint. Using the new "M365D" provider name will cause the provider to use the new combined Microsoft Defender APIs, giving you consolidated access to data from multiple Defender services.

To create a provider for Microsoft 365 Defender: python m365d_provider = QueryProvider("M365D") m365d_provider.connect()

All template queries for that were available for MDE are also loaded for M365D.

Using Microsoft Defender for Endpoint is still supported. To use only the Microsoft Defender for Endpoint APIs use "MDE".

You can specify your connection parameters in msticpconfig.yaml yaml DataProviders: # ... MicrosoftDefender: Args: ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa ClientSecret: KeyVault: TenantId: 8360dd21-0294-4240-9128-89611f415c53 This configuration expects to find the client secret in your configured Key Vault with the name "DataProviders-MicrosoftDefender-Args-ClientSecret" - the msticpy config editor will create this for you. You can store the secret in your config file but, obviously, we would not recommend this.

You can also specify configuration/secret values in the call to m365d_provider.connect(). For more information see the Defender provider MSTICPy documentation

Other improvements

The Graph provider now supports sovereign clouds as well as the Azure global cloud. You can configure settings for MS Defender and MS Graph via the MSTICPy settings editor. If you have configured settings for these you can now connect with a simplified syntax, with no parameters required. python graph_prov = QueryProvider("SecurityGraph") graph_prov .connect()

The schema for the configuration is the same format as the Defender format described above.

yaml DataProviders: # ... MicrosoftGraph: Args: ClientId: 66b9818a-26cd-4584-8eb0-7f7a499242aa ClientSecret: KeyVault: TenantId: 8360dd21-0294-4240-9128-89611f415c53

Process Tree visualization supports MS Sentinel schema for MDE process data

The schema of the DeviceProcessEvents table in Microsoft Sentinel is different from the data queried directly from the MS Defender advance hunting API. This update allows you to query MS Defender process data from Azure Sentinel and display in the Process Tree visualization.

New CI Pipelines using GitHub Actions and Azure Pipelines

The old Azure DevOps pipeline has been decommissioned and replaced by two new ones - GitHub actions and an improved version of the Azure DevOps pipeline.

The main advantage of the GitHub actions pipeline is that the results of tests and code checks are visible to external contributors.

However, there are some component governance and security checks that we are required to run by Microsoft but which are not currently available in GitHub actions. We've opted to run both in parallel.

The new Azure DevOps pipeline is more sophisticated and comprehensive than the previous one. It allows us to run builds and tests on Windows, Linux and Mac platforms and to cover Python versions 3.6 and 3.8. (Note: we will likely switch to 3.8 and 3.10 in the near future)

Other changes and fixes

98864aa@Ianhelle/kusto and md365 support 2021 11 01 (#225) - Fixing issue in odatadriver.py where apiroot would keep getting suffix appended if you connected multiple times. - Adding mutli-cloud support for MS Graph API in securitygraphdriver.py - Fixing syntax errors in graphalerts.yaml - Updating tests for testdrivers and testkqldriver - Fixed an issue in kqldriver with Kusto returning a different status code format 1c2c1be@Disabling IPstack tests because of restricted API allowance (#224)* - Disabling IPstack tests because of restricted API allowance - Fixing incorrect assignment in testgeoip.py - 56fcb81@Updating readme badge to reflect new build pipeline - cdff1ff@Update sphinx requirements in azure-pipelines.yml for Azure Pipelines - ce8cbe1@Update README.md to include downloads - 0cb20fd@Updated way of getting instance and domain name for azuremltools.py (#217) - Updated checking for version of msticpy using setuptools/pkgresources in azuremltools.py - Added warning if running with stale import of newly updated msticpy. - 8043207@Ianhelle/timeline args check 2020 09 21 (#219)* Adding some error checking of column names to timelinevalues and timelineduration - a7498a3@Ianhelle/azure pipelines yaml 2021 09 09 (#201)* - Pylint and mypy fixes in: - anomaloussequence/model.py - anomaloussequence/utils/cmdsparamsvalues.py - common/pkgconfig.py - common/wsconfig.py - config/compedit.py - config/mpconfigfile.py - data/dataproviders.py - data/dataqueryreader.py - data/querystore.py - data/sqltokql.py - datamodel/entities/host.py - datamodel/entities/ipaddress.py - datamodel/pivotregisterreader.py - nbtools/morphcharts.py - nbtools/nbdisplay.py - nbtools/securityalert.py - nbtools/securitybase.py - sectools/auditdextract.py - sectools/base64unpack.py - sectools/cmdline.cmdline.py - Adding pylint --disable=duplicate-code - Updating pylint requirements in requirements-dev.txt and .pre-commit-config.yaml - Add [withpyroma] to prospector install - Duplicating @noqa MC0001 suppressions since prospector/McCabe now seem to ignore decorators an locate the offending line as the "def" line rather than the decorator line. - Fixing MyPy-discovered error in nbwidgets.AlertSelector 9f122ee@refactored code in testgeoip 907e486@Remove un-needed code in testgeoip 8ae01f7@Updated testgeoip to fix failure case e5aadac@Added missing parameters to listallsigninsgeo

- Python
Published by ianhelle over 4 years ago

msticpy - Lazy loading of Config Editor tabs

Minor release to address some performance problems with the config editor on Azure ML notebooks platform. The control uses lots of ipywidgets and the AML notebook front-end has some perf issues with lots of widgets. The improvement is to not load all controls/widgets for the settings editor - only load each tab as it is selected.

The release also includes a minor change to the entities classes - to allow them to be JSON serializable by adding a JSON Encoder class to entities.Entity. python json.dumps(my_entity_list, cls=entities.Entity.JSONEncoder)

- Python
Published by ianhelle over 4 years ago

msticpy - Azure CLI session check

Minor release that implements check for existing Azure CLI credentials and warns user if these are not valid/expired.

Pull request #209 Azure CLI credentials check

- Python
Published by ianhelle over 4 years ago

msticpy - Hotfix++ Fixing settings editor not saving values.

I introduced a bug in the previous hotfix that caused the settings editor to ignore changes made. This should be fixed along with a couple of other related items.

- Python
Published by ianhelle almost 5 years ago

msticpy - Hotfix - Azure cloud settings fix

Minor fix to cope with incomplete configuration in Azure settings.

- Python
Published by ianhelle almost 5 years ago

msticpy - 1.4.0 Hotfix

Release details

Some of the features scheduled to go into the 1.4.0 release did not get published correctly. 1.4.1 should fix this. It also includes some updates to fix problems in the Process Tree visualization.

Fixes

b2b7fd8@Bug fixes for process_tree.py and related modules. Fixed: - not working with generic data input. - not displaying command line correctly - not display time in Hover tool correctly - not truncating command line for display correctly - missing params in doc string - removed dependency on Linux auditd schema for converting numeric fields to strings

- Python
Published by ianhelle almost 5 years ago

msticpy - Azure cloud, Matrix plot and ProcessTree for MDE

Highlights

We've added comprehensive support for Azure sovereign clouds (China, Germany and US Government). To set the appropriate cloud open MpConfigEdit and choose the Azure tab.

Azure cloudt Read more about this here

There are also additions and improvements to the visualizations.

Matrix Plot is an interaction plot letting you visualize the interactions between two sets of entities. A typical use would be seeing connections between source and destination IP Addresses.

Matrix plot

New Features

889112d@Matrix plot visualization (#197)* Matrix plot visualization Includes - vis/matrixplot.py - main plotting code - MatrixPlot.ipynb - notebook - visualization/MatrixPlot.rst - documentation - tests/vis/testmatrixplot.py - unit test - mppandasplot - generic pandas extension "mpplot" that includes all visualizations

d425c77@Process tree updates to handle more generic data (#178)* - Tidying up some of the code in timeline.py - Moving the pd accessors to timelinepdaccessor.py - Updating notebook EventTimeline.ipynb and documentation EventTimeline.rst - Unit tests for timeline controls in testtimeline.py - Removed line to slim down columns - need to keep original data columns because they may be referenced in sourcecolumns, etc. - Process tree updates for MDE process data - Updating docs, adding pid_fmt parameter - Adding explanation of schema to NB and doc - Adding updated notebook - Expanded the schema table to show all types.

a00741d@Adding support for multiple Azure Clouds (#193) Co-authored-by: Pete Bryan pebryan@microsoft.com - Adding cloud options to settings, kqldriver and keyvault - Updated help text and combined CESimpleSettings class. - Azure auth multicloud support - Replacing hard-coded URIs with msrestazure enums - Modularize multicloud support - Refactoring code to use AzureCloudConfig instead of reading directly from config - azureauth.py - uses AzureCloudConfig, added credentialscopes option SubscriptionClient init, spelling corrections - azureauthcore.py - create AzureCloudConfig, and _AzCachedConnect classes to read cloud config values and cache tokens - credwrapper.py - minor formatting - keyvaultsettings.py - use AzureCloudConfig - cecommon.py - uses AzureCloudConfig in getdeftenantid function - azuredata.py - uses AzureCloudConfig, added credentialscopes option to client class inits, spelling corrections - azuresentinel.py - uses AzureCloudConfig. added functions to set default subscription and workspace, changed buildpaths to method - kqldriver.py - uses AzureCloudConfig, added loadmode == Silent, added mpazauth to control using Azure auth instead of native Kqlmagic auth, - resourcegraphdriver.py - uses AzureCloudConfig, added credentialscopes option SubscriptionClient init - testazureauthcore - Added test for AzureCloudConfig class - testresourcegraphdriver.py, testazuredata.py, testazuresentinel.py - added real credential classes to test - msticpyconfig-test.yaml, msticpyconfig.yaml - added Azure sections to test config. - Update azure-mgmt-subscription and azuremgmt-resource versions

Fixes

d81801d@Ianhelle/tests and tidy 2021 09 01 (#200)* - Added separate unit test for nbwidgets to cover cases missed in notebook. - Removing deprecated code from utility.py - Removing unneeded function from mordordriver.py - Sourcery code refactor in entity.py - Deprecating createiprecord function in iputils.py - Split wsconfig tests out of testpkgconfig.py - Adding testwsconfig.py. Added new test for couple of utility methods - Removing redundant import from testnbwidgets.py - Moved lots of test files into sub-package folders corresponding to tested modules - Merged a few tests into single files - testentity.py and testtimeline.py - Re-enabling IPstack/geoip notebook test in testgeoip.py - Updating use of tqdm to remove deprecated tqdmnotebook in utility.py - Adding some debugging output to geoip GeoLite class. - Updating testgeoip.py to use predictable msticpyconfig-test.yaml - Updating test to restore env var values after test in testpkgconfig.py - Fixing aliases element in kql query records - Handle aliases as list or string in help * Updates from PR review

f58e35c@Changing behavior so that vtlookupv3.py does not throw an exception when an item is not found. (#199) - This is especially annoying when doing multiple look-ups and one item is not found. - Minor changes to VTLookupV3.ipynb - to use get_object API from the vtlookupv3 library

d425c77@Process tree updates to handle more generic data (#178) - Fixing bug in customjs and adding mde test files - Update NOTICE.txt - Fixing progress widget to use correct visibility settings. - Changing userconfig to be more defensive against bad settings. - Error in querystore passing list instead of strings to MsticpyUserConfigError exception

Adding a bit more text/warning to check the browser.

Co-authored-by: Pete Bryan pebryan@microsoft.com ff87ebb@cred-scan suppression (#196)

- Python
Published by ianhelle almost 5 years ago

msticpy - Kql from Python Scripts, Mordor provider fixes

New Features

We've updated the Kql/Azure Sentinel data provider so that it no longer depends on IPython/Jupyter functionality to work. This prevented using the provider from Python scripts and applications. The new implementation uses Kqlmagic as a standard import and no longer requires IPython magics to work.

08c7cf1@Ianhelle/kql as function 2021 08 04 (#190) Kqlmagic operations called as func vs. IPython magics Now using Kqlmagic.kql() function in kql_driver, allowing it to be used from python code.

Fixes

The main fix here was to adapt to changes in the Mordor attack data repository Mordor is now known as OTRF Security Datasets (Although it will continue to be known as Mordor in MSTICPy for the time being).

17f9704@Ianhelle/modor url refix 2021 08 04 (#189)
fabde53@fix img src for broken preview images (#191)
17b9227@Pebryan/7 16 21 sent api update (#187)* added incident collection
- Updated pandas version to pandas>=1.1.5
- Updated docstrings for Azure and Azure Sentinel APIs

Potentially breaking changes

MSTICPY now requires pandas 1.1.5 as a minimum

- Python
Published by ianhelle almost 5 years ago

msticpy - Timeline duration, MDE Query upgrade

New Features

Timeline duration visualization #176

This new variant of the timeline control allows you to group by one or more columns and view timelines as start/end of activity related to those columns. The graphic below shows durations of communication between different source and destination IPs using a given protocol.

Like other timeline controls it's accessible as a pandas extension.

python az_net_flows_df.mp_timeline.plot_duration( group_by=["SrcIP", "DestIP", "L7Protocol"] )

Timeline duration

You can also import and use the plot function directly:

```python from msticpy.nbtools.timelineduration import displaytimeline_duration

displaytimelineduration( hostlogons, groupby="Account", refevents=hostlogons.sample(3), ref_col="TargetUserName", ); ```

Read about this in our Event Timeline documentation

Fixes

184 MDEFix Update MDE connectors

The Microsoft Defender for Endpoint provider and queries have had a major update, correcting many schema issues (we were still using old column names in a number of cases

185 from microsoft/ianhelle/initnotebookchanges-2021-07-14

Removed nbcheck.py update from azureml_tools.py
Updated text about how to fix missing PyGObject
Fixed MpConfigEdit widget to show checkbox for making backups prior to updating msticpyconfig
Fix to remove casefold of OptionButtons value

8707868@Fixing Mordor and azuremltools tests - Fixed Mordor URL for mordor data provider

1ec1a00@bypassing GeoIP notebook test because of IPStack intermittent problems (#181)

- Python
Published by ianhelle almost 5 years ago

msticpy - Hotfix for notebooks #2 - Pivot exceptions and nbinit

Fixes

788b4ee@ - Added new context manager in MsticpyUserError to stop standard display of msticpy user exceptions. Note this doesn't suppress the exception, just stops exception being output to notebook even if exception is caught.
Example: python from msticpy.common.exceptions import MsticpyUserError with MsticpyUserError.no_display_exceptions(): # do stuff that may cause exceptions to be display.

Fixed typo in path for msticpyconfig.yaml was causing searches to always fail.
Updated testnbinit.py - fix one test. Using temppath instead of tmpdir fixture.

- Python
Published by ianhelle almost 5 years ago

msticpy - Hotfix for Azure Sentinel notebooks

A few hotfixes for bugs

33e9fd4@Fixes to dataproviders, data view, pivots and nbinit - Replace list.pop with list.remove in dataproviders.py - caused an error when using provider internal querytime control to set query start and end params. - Add exception catching around Pivot instantiations in pivotregisterreader.py to prevent errors when loading Pivots with some providers missing configuration settings. - Prevented pushnotebook from being called before displaying control - this was causing an error when first displaying the DataViewer control - Removed pandas option to return html schema in nbinit.py - Azure ML notebook dataviewer throws a react exception when trying to display DataFrame columns with complex types (e.g. dict or list vs. string/int/etc). Turning this off prevents the AML viewer being used by default.

- Python
Published by ianhelle almost 5 years ago

msticpy - Dataview, Azure Resource Graph and Sumologic driver

Highlights

The highlights of this release (which is really 1.2.0 with some last-minute fixes) are: two new data providers for Azure Resource Graph and Sumologic and a DataViewer control for pandas dataframes.

Azure Resource Graph provider

The Azure Resource graph provider lets you query Azure resources using KQL queries. This works much like our other query providers and gives you a lot more flexibility in navigating around Azure resources than the current AzureData provider.

Explore more in the Resource Graph Driver notebook and the online docs

Many thanks to Ryan Cobb @rcobb-scwx for creating and contributing this.

Sumologic provider

Sumo Logic Inc. , is a cloud-based machine data analytics service focusing on security, operations and BI use cases. This provider allows you to connect to and query your data from MSTICPy via their Search API Explore more in the Sumologic Data Connector notebook

Many thanks to Julien (@juju4) for building and contributing this.

DataViewer

The data viewer uses the [Bokeh DataTable control[(https://docs.bokeh.org/en/latest/docs/user_guide/interaction/widgets.html#datatable) to display and browse through data in a pandas DataFrame. It lets you sort by column, choose which columns to display and filter by multiple columns. It keeps a synchronized copy of the DataFrame with column choice and filtering applied, so that you can always access the data as it appears in the control.

Explore more in the DataViewer notebook and the Dataview online docs

New Features and Updates

ab6eb73 - (#164) DataViewer control

DataViewer control - Bokeh based data table control
Documentation in DataViewer.rst
Notebook in DataViewer.ipynb

3ab31ad- Azure Resource Graph Data Provider (#167)

Co-authored-by: Ryan Cobb rcobb@secureworks.com
Co-authored-by: Pete Bryan peter.bryan@microsoft.com

2a33d61- Sumologic Data Provider (#165)

Co-authored-by: juju4 juju4@users.noreply.github.com
Co-authored-by: Ian Hellen ianhelle@microsoft.com

4c4b8ca- Ianhelle/getting started fixes 2021 06 04 (#170)

Updates to initnotebook in nbinit.py
- Moved most of the logic of the Azure Sentinel notebooks nbcheck script into msticpy - azureml_tools
- Will now create msticpyconfig.yaml if it doesn't exist
- Output message from init_notebook when msticpyconfig.yaml is created in nbinit.py

8214c90 - Minor fixes to VT TI Provider and TIBrowser (#173)

Adding QueryTime instance to Dataproviders in data-data_providers.py. Let's you specify time range (or use default 1 day range) for all queries from provider
Added Browshot to mpconfig_defaults.yaml so that it can be edited in MP settings editor

ab6eb73 (#164)

added recent blog articles and video to a renamed resources page for RTD - blob_articles.

Fixes

8214c90 - Minor fixes to VT TI Provider and TIBrowser (#173)

Updating VT TI Provider for better severity accuracy with additional "detected_" types.
Changing ti_browser to display "information" results if no "warning" or "high" items in dataset.
Updated compoundctrls.py and mpconfig_file.py to work when azure-keyvault-secrets not installed
Update to QueryTime to fix default max time when "day" is used for units.

e6ea232- Pebryan/2021 6 7 grey noise updates (#171)

Update GreyNoise naming and added custom UA

4c4b8ca- Ianhelle/getting started fixes 2021 06 04 (#170)

Fixing a query error in kqlsentazuresentinel.yaml
Fixing an issue with contain naming when creating pivot functions
Fixing link in TIProviders.rst
Fixing mypy error in azureauthcore
Adding check for KV settings before trying to initialize secrets client in provider_settings.py
Adding function to search for file in utility.py
mpconfigedit.py:
- Renaming "Save File" button to "Save Settings"
- Add config_filepath parameter to specify file to save when settings not imported from a file
- Changing item update button to "Update" rather than Save in comp_edit.py
- Strip extraneous quotes from string "default" value in mpconfigcontrol.py
- Corrected open params to "w" rather than "w+" (write append)
Added "browse" as alias for browsequeries in dataproviders.py.
Fixed misreporting of DataFrame column name in exception when column parameter does not match a column in the input data. in pivot_register.py
Adding azuremltools.py (from nb_check.py script)
Added "severities='all'" parameter option. Added friendly warning (rather than exception) when no data is present in the input (either empty DF or nothing above the default severities - in ti_browser.py
Fixing additional areas breaking in timeline plot module if you don't supply "source_columns" parameter.
Fixing mypy warning in domain_utils.py
Fixed allowing geolite to create a database folder even if parent folders don't exist.
Adding "col" and "column" aliases for lookup_iocs in tilookup.py
Changed score logic for IBM XForce - prevent misreporting of score==1 as warning.
Updating version in _version.py
Added testazureml_tools.py
unittestlib updates:
- Added change_directory context manager (with lock) to allow individual test cases to change directory
- Fixed TESTDATAPATH to always return absolute path
Fixing pylint warnings about the lack of use of context managers in mordordriver.py, base64unpack.py, morphcharts.py
Pylint warnings suppression about lowercase enums in tilookup.py
Fixing version number format problem in azuremltools
Fixing warning that wasn't processing an f-string in tilookup.py

- Python
Published by ianhelle about 5 years ago

msticpy - Pivot functions #3 and Greynoise TI Provider

This minor release includes some major updates to the pivot functions. It also includes the addition of the Greynoise Community API to our set of Threat Intel providers. Also

Warning - this includes some potentially breaking changes since we have renamed a number of pivot functions to have shorter, friendlier names. In particular a number of Azure Sentinel query functions have shortened names.

Updates

Pivot phase 3 with better joins, more pd accessor functions and short/friendly aliases for pivot functions (especially queries) #140 - Added short names for many queries - Adding shortcut functions to entities - these allow you to create links from pivot functions in containers e.g. IpAddress.whois() -> IpAddress.util.whois() The shortcut methods have the advantage of being easier to see and type in. They have the further benefit of working as instance methods. So, given an an instance you can do this: python my_ip = IpAddress(Address="157.23.49.1") my_ip.whois() # previously you'd have to do IpAddress.util.whois(my_ip) - this still works - Standard pivot functions now also support joins (previously these were implemented only for Data query and TI functions) - joins now support join_ignore_case parameter to join case-insensitively - Added PD accessors for: - filter_cols - column filtering with patterns or lists of columns - filter - string or regex will match against all columns (can also match on partial numbers with the numeric_col=True option) - sort - sort on one or more columns or a pattern to match columns - list_to_rows - expand embedded lists in one or more cols to rows - parse_json - turn JSON strings in column into Python objects - Updates to Pivot docs and notebooks for new pivot behavior - Added removepivotfuncs method to pivot.pivot.py + unit test in test_pivot

Implemented Greynoise API as TI Provider - Greynoise Community API - TI Docs

Entities - Added additional entities from schema: iotdevice, mailcluster, mailmessage, mailbox, submissionmail - Adding str and repr to Edge in entitygraph.py - Adding (not yet used) graphproperty.py to handle automatic creation of edges

Other updates - Added tilookup functions to allow enabling/disabling individual providers - Add better timestamp conversion in vtlookup3.py. - Added check for attribute name similarity in querycontainer - if trying to access an attribute that doesn't exist it will return likely matches in the exception - Extracted dup code from pivotdataqueries to pivotregister. - Updating blog articles links in docs with recent publications - Capturing output from loaduserdefaults in nbinit.py to display as HTML - Adding findanomalyperiods function and shorter alias for timeseriesanomaliesstl in timeseries.py

Fixes

Fixed a problem with logon failure query parameters.
Minor corrections to queries in kqlsentaznetwork and kqlsent_azure
Fixing path handling in file.py entity
Fixing test in testpivotpd_accessor for renamed parameter.
Updating notebooks for some pivot function renaming.
Fix spelling errors in SettingsEditor.rst
Remove unnecessary list comprehension in testsecurityalert.py
Add tests for listtorows and parsejson in pivotpdaccessor.py - testpivotpdaccessor.py
Fixed some errors in entities with description_str and identity fields
Add missing FullName property to Host
Fixed exception in IpAddress
Fixing some test errors in testmordordriver (due to test file race conditions)
Removing duplicate mp_config file
Correcting problem with passing namespace argument to notebooklets in user_config.py
Fixes to analyzeimports.py and importanalyzer.py that prevented use from commandline and use with other packages.
Fix to listtorows pd accessor in pivotpdaccessor.
Changed timeline.py so that it doesn't error when no source_columns are supplied
Fixing bug in calculating period in timespan.py
Cosmetic fixes in eventcluster.py
Fixing an issue in ti_browser if passed an empty dataframe.
Updating requirements files for updated versions of lxml and aiohttp for security issues.

- Python
Published by ianhelle about 5 years ago

msticpy - Bug fix for pip install --no-binary and Mitre downloads

Fixes

Using pip install --no-binary fails because requirements.txt, requirements-dev.txt and Readme.md are not included in the package. These are referenced by setup.py when using the --no-binary option
Changed download of Mitre techniques and tactices from web scraping to official JSON. Fixes error in Mordor driver

- Python
Published by ianhelle about 5 years ago

msticpy - MSTICPy v1.0.0

Overview

We've finally decided that we're ready to put a 1.0.0 release number on MSTICPy and move out of the Beta world.

There are several new features in this release and some significant updates to other features.

New:

Flexible dependencies with pip extras
Settings management and auto-load of components
An experimental SQL to KQL translator

Update:

Significant updates to the pivoting library.
Updates to notebook initialization include Azure CLI single sign-on and Azure ML data explorer.

Semantic versioning

Although we've been a little "flexible" with our use of version numbers while in beta, we plan to adopt standard practice for semantic versioning moving forward:

Updates to major release number will indicate significant breaking changes
Minor release number increments will indicate significant new or changed features
Point/patch updates will indicate fixes and minor feature improvements

E.g. version Major.Minor.Patch

New Features

MSTICPy Dependencies - Implementing Extras #128

MSTICPy relies on a number of Python packages such as pandas, numpy and bokeh. Previously, many rarely-used dependencies were installed regardless of whether they were going to be used or not. We've now split MSTICPy dependencies into "extras". The base install of MSTICPy only installs a core set of dependencies - others can be installed on demand by specifying the name of the extra feature that you want to use.

This is coupled with a refactoring of the code so that, should you try to use functionality that does not have a required dependency installed, an informative exception message is displayed telling you which extra you need to install.

In this release we are using KqlmagicCustom (still in pre-release). This also uses "extras" and has a significantly smaller dependency list for its base install. The switch to KqlmagicCustom alone has halved the MSTICPy dependency set for a core installation. The install time should be reduced from several minutes to a few seconds in typical usage.

Feature details

Implemented pip "extras" for msticpy install - drastically reduces install time for core msticpy.
Refactored many modules to emit informative warning if user tries to load functionality that requires a different "extra"
Refactored unit tests to work with missing extras.
Added extras documentation to Installing documentation
Added requirements-all.txt - that will always install all dependencies.
Added pre-commit hook to generate requirements-all.txt

Documenation

MSTICPY configuration settings management #136

The MSTICPy configuration file has grown to be quite complex. We've tried to address this by creating some interactive tools to let you create and edit settings using a simple GUI. We've also created a notebook that walks you through creating your settings file for the first time. The Getting Started and ConfiguringYourEnvironment notebooks in the Azure Sentinel Notebooks GitHub repo have also been updated to use these tools.

MSTICPy itself, has a number of initialization/loading steps that you need to carry out before starting to use it in a notebook. The other part of this feature is the ability to specify, in your settings, what components to load at initialization. Components that can be auto-loaded include: DataProviders, TI Providers, Notebooklets and Pivot functions. These are specified in the msticpyconfig.yaml (you can edit these settings with the Settings editor) and are auto-loaded when you run init_notebook() at the start of your notebook.

Feature details

Flexible UI for configuring MSTICPy settings
User environment configuration for notebooks - lets you specify (in settings), which providers/modules, etc. that should be loaded automatically.
Added HTML output from nbinit to show imported modules
Added check_version in init_notebook function to indicate if a new version of MSTICPy has been released.
Added a function to retrieve and show current KV secrets
Updated MSTICPy Configuration documentation
Added MSTICPy Settings Editor documenation
Added MSTICPy Settings notebook
Added documentation diagram Config editor, auto-load, dependencies and Sql2Kql translator

Documentation

SQL to KQL Translator #132

This is a simple (and somewhat experimental) feature to convert ANSI SQL to KQL queries.

Core feature implemented as msticpy.data.sql_to_kql using moz_sql_parser
Support for SELECT, WHERE, JOIN, UNION, GROUP BY and others.
Support for limited SparkSQL extensions
SQL TO KQL Conversion Documentation
SQLToKql Notebook

Documentation

Updated Features

Pivot Functions #131

The pivot functionality has been updated with several usability and documentation improvements. Notable features include:

Pivot browser, letting you browse/search for pivot functions.
Persistent Pipelines - you can save pipelined pivot operations in a YAML file and run them on input DataFrames

Note: Shortly after the MSTICPy v1.0.0 release, we will be releasing another update to Pivot Functions that will (among other things) shorten pivot function names to be friendlier and easy to type. If you are starting to use Pivot functions, please bear this in mind when testing/coding.

Full list of changes

Made AzureSentinel and MDE the preferred names for LogAnalytics and MDE drivers.
Added pivotbrowser UI - pivotbrowser.py
Added ability to read pipeline definitions from yaml files - pivot_pipeline.py
Adding pivot.teeexec pipeline function - in pivotpd_accessor.py
Add ability to add arbitrary/ad hoc functions as pivots - in pivot.py
Exposing get_timespan function in Pivot class as public function - in pivot.py.
Added DNS entity to several pivot functions - mppivotreg.yaml
Fixed some queries for more consistency.
Pivot data query functions now prefixed with table name.
Added ability for pivot functions to return raw output.
Add joins for pivot data queries in pivotdataqueries.py
Add "print" query debug parameter in data_providers.py
Add findentity function in entities _init__.py
Add "pivots" attribute (an alias for getpivotlist) in entity.py
Add ability to set timespan more flexibly. Calling set_timespan() no longer resets the timespan.
Add PivotBrowser method to Pivot class - in pivot.py
Switched engine to "Python" for pd.readcsv in pivotmagic_core.py to handle more formatting types.
Add positional parameters to pipeline step and cleaned up code in pivot_pipeline.py
Updated PivotFunctions documentation
Updated PivotFunctions Notebook
Added PivotFunctions-Introduction notebook

Documentation

Notebook Initialization

MSTICPy has a module nbinit and we use the main function in that module (init_notebook) to streamline a lot of the setup needed to get a notebook up and running. It does things like: import modules, install required packages, check configuration.

We've made a few significant updates to this module including:

Defaulting to using the native data browser in Azure ML notebooks
Tidying up and minimizing output so that warnings are clearer and help URLs are displayed clearly
Enabling Single sign-on for notebooks using Azure CLI (all notebooks in a session will use the Azure CLI authenticated session to obtain tokens for your Azure Sentinel workspaces)

Miscellaneous Updates and Additions

9a8d648 (#149)
- Adding time unit control to QueryTime widget to allow interactive setting of day/week/month. Also increased the max range for these units - in nbwidgets.py
- Docs: Updated notebooksamples.rst with latest notebooks. https://msticpy.readthedocs.io/en/latest/notebooksamples.html
- Docs: Added Releases.rst - pointer to GitHub release page. https://msticpy.readthedocs.io/en/latest/Releases.html
- Docs: Generated new DataQueries.rst doc https://msticpy.readthedocs.io/en/latest/data_acquisition/DataQueries.html
- Docs: Added links to medium articles in ReadtheDocs https://msticpy.readthedocs.io/en/latest/blog_articles.html
- Docs: Added SplunkProvider documentation
- Docs: Updated README and package summary section of MSTICPy to align with current features
- Reordered parameters in wsconfig.py WorkspaceConfig so that you can supply the workspace name as single positional parameter
python ws_config = WorkspaceConfig("MyWorkspace") - Change kqldriver.py so that you can supply a WorkspaceConfig instance as the "connectionstr"

python qry_prov.connect(WorkspaceConfig("MyWorkspace")) - Change to pivot_register to return single "raw" result if it is a list of one item - Add Pivot and entities as auto-imported items - Add create static method to entity to instantiate entity from dict or pd.Series to allow easier programmatic creation of entities.

python acct = Entity.create({"Name": "ian", "Type": "Account"}) - Adding more Azure Sentinel queries for VMComputer and DNSEvents tables. - nbinit.init_notebook(): - Added resource URLs to warnings - added KQLMAGIC_CONFIGURATION to enable trying AzureCLI SSO by default. - added pandas config to return schema with dataframe html to render using native nteract data browser.
cbe918e
- geoip.py - Adding more descriptive to text on how to fix things in exception if no API keyvault_settings
- tilookup.py - added raise MsticpyUserConfig exception with help URIs if no providers are enabled - in lookupioc, lookupiocs
4c32ab9
- Changed URL formatting so that URL help links in MSTICPy exceptions use about="_blank" to open in new tab - in exceptions.py.
- Switched to using a list for output instead of concatenated string in MsticpyUserError._repr_html_()
- Tidied up formatting of text and updated exception URL in wsconfig.py
- nbinit.py:
- Add about="_blank" to URLs in warnings,
- Tidied up some text
- Changed red error to orange warning
- Trapped seaborn import error if not installed
- Print out list of imported packages by default
- Changed Development Status classifier to 'Production' and adding some extra keywords in setup.py
- Replaced matplotlib draw_entity_alert_graph with Bokeh version in nbdisplay.py
f386ccd (#136)
- DataMasking:
- Adding hashaccount as separate item type to dataobfus.py
- Making hash_ip more flexible - ignoring things like localhost
- Updating documentation, tests and mapping file.
1cebe8c (#131)
- Made 'AzureSentinel' and 'MDE' the preferred names for LogAnalytics and MDE drivers: ws_config = WorkspaceConfig("AzureSentinel")
- Update MordorData.rst doc with better intro section

Fixes

0b7d1dc Updating KqlmagicCustom version
48f02b5
- Handle cases where empty data set is passed to timeline functions - timeline.py
- Fixing occasional race condition in testing with KeyringClient.iskeyringavailable
5915b9d
- Adding MC0001 McCabe suppression to nbinit.py
- Fixing typo in DF name in syslog_utils.py
f01d094 Updating version to 1.0.0
- Fixing testsecurityalert.py unit test
- Adding small test for KeyringClient - testprovidersecrets.py
c4c1fa2
- Linux bug - Error loading secretsettings in environment that doesn't have a supported Keyring backend. Added test to KeyringClient before trying to load. Also fixed logic to properly honor settings (subject to above check) - previously the `usekeyring` param defaulted to True so would try to load Keyring even if settings were = False.
bd99b11
- If running in IPython environment checkandinstallmissingpackages() will use %pip IPython magic function rather than subprocess in utility.py. This ensures that the pip install will end up in the same Python environment that the notebook kernel is running in.
- Suppressing warnings in Kqlmagic load in kql_driver.py
- File entity "FullPath" generation now handles None values for directory and separator more gracefully in file.py
- Changed color attribute extraction for draw_alert_entity_graph and plot_entity_graph so that it defaults to a color even if the node has no color attribute - in nbdisplay.py
- Handle inter-entity references in child entities in security_alert.py (e.g. if entity1[$id=1] has entity2[$id=2]) nested inside it and other entities have references ($ref=2) to entity2
- Handle cases where no "Name" attribute is defined in account entities.
- Added additional test cases for nested entity references in testsecurityalert.py
bee0f26, 69256bf, cbe918e
- If any warning from WorkspaceConfig, print this out - nbinit.py
- Added additional extra for KqlmagicCustom to install pyperclip
- Changed azconnect defaults to all authmethods if None supplied in params or config - azure_auth.py
- azureauthcore.py - added default_auth_methods() function
- keyvaultsettings.py - default to using all authmethods
- provider_settings.py - do not instantiate SecretSettings unless KeyVault config has settings
- wsconfig.py:
- Changed wording of warnings/error messages
- Does not now raise an exception if no suitable config found
- Does not raise a Python warning if we rely on searching to find a config file.
- Changing default entity graph plot size in draw_entity_alert_graph/plot_entity_graph shows the plot by default unless hide=True parameter - nbdisplay.py
- nbinit.py
- Changed all output so that it is HTML text vs. print
- Captured output from called functions that print output
- Minor rewording and changing header size of titles.
- securityalertgraph.py - check if File entity has FullPath attribute before accessing it.
- Added mozsqlparser back to mypy.ini
- Changed testpkgconfig.py so that it gives clearer message when test fails
- added "KqlmagicCustom[jupyter-extended]" to conda exceptions
- Updated testpkgconfig.py to account for WorkspaceConfig no longer produces and error.
9a8d648 (#149)
- Updated msticpy notebooks
- Updated setup.py, requirements.txt and conda-reqs with new/non-conflicting version
- Fix to testnbinit.py and importanalyzer.py
- Added test script for dependencies/extras
- Updating version to pre3/rc3
- Fix to README
- Removed random unicode char in PivotFunctions.rst
- Fixing test error in test_nbinit.py
- Some fixes to notebooks (mainly getting rid of unneeded Seaborn refs)
- Doc string error - prospector - in testmpextras.py
- Fixed QueryProviderDocumenter.ipynb
- Added KqlmagicCustom[jupyter-basic] to core components and Updating docs for this in Installing.rst. Updated testpkgimports.py to account for this.
- Setting KQLMAGICEXTRASREQUIRE env var in package init.py to prevent warnings in Kqlmagic
- Minor clarification in MPSettingsEditor.ipynb
- Fix bug in ip_utils.py convert to entities
- Fix to provider name handling in user_config.py
- Fixing wording in SelectAlert widget
- Fixing potential None value error in GetText and GetEnvironmentText
- Fixing a few bugs and tidying in convert_to_ip_entities in ip_utils.py
- Fixing issue #146 - Error is thrown when AzureSentinel config is not present in msticpyconfig.yaml file
- Added extra tests for nbinit.py in test_nbinit.py
- Fixing Issue #148 infinite recursion issue in processtreeutils.py
- Adding file lock to unittestlib::custommpconfig - because settings are global, multiple test processes can change the settings on each other.
- Minor updates to test_nbinit.py
- Fix in pivotregister.py - `iteratefunc` did not pass through **kwargs to function. Also added a few comments to explain what's going on.
- Bug in securityalertgraph - if NTDomain attribute is None
- Added local function cache to better handle repeated IPs
- Missing update to all_ips perf optimization in convert_to_ip_entities
- Adding filelock to dev requirements-dev.txt
- Corrected pip extras syntax in Installing.rst
- Invalid return type in pkgconfig.py:: validateconfig
- Added prompt_for_ws function to wsconfig.py. Also added more detail to warnings on how to fix things.
- Adding markdown and beautifulsoup4 to requirements-dev.txt and conda-reqs-dev.txt
- Fixing test error - seems like subtle change or bug in pandas groupby behavior - in sessionize.py
- Adding more verbose output to try to catch spurious errors in test_nbinit.py
- Workarounds for some test errors in testnbinit.py and testuser_config.py
- Updating requirements-dev.txt with pip-compatible versions.
- Adding same updates to conda-reqs-dev.txt and conda-reqs-dev-pip.txt
- Error fixed in initializing notebooklets and pivots in user_config
- Making test more flexible in testiputils.py to allow for IP address moving ASNs.
1cebe8c (#131)
- Remove unneeded code from keyvault_client.py
- Fixed pivotregisterreader to skip classes that cannot be instantiated (e.g. IPStack if user doesn't have API key)
f386ccd (#136)
- Correction to FoliumMap.ipynb, removing dropna from read_csv in FoliumMap.ipynb
- fix to config2kv.py to correct some problems, Also added a function to retrieve and show current KV secrets
- fix for ipwidgets warning about deprecated on_submit() method
- multiple fixes for typos and duplicate section names in: DataProviders.rst, UploadData.rst, PivotFunctions.rst
- added SplunkProvider.rst doc for Splunk provider
- fixed issue in nbinit.py where extra_imports were being lost.
- fix for QueryTime in nbwidgets.py - exception if user types invalid value into date field.
- fixed several issues in testmprelease.cmd with messed up folders/current folder.
efccf19 (#137)
- Couple of minor fixes to txt2df magic.

- Python
Published by ianhelle about 5 years ago

msticpy - MSTICPy 1.0.0 RC4

Overview

The work in this release is mostly bug fixes and usability improvements Pull request for all changes is #147

New Features

20db6cb@Added promptforws function to wsconfig.py. Also added more detail to warnings on how to fix things. python ws_config.prompt_for_ws()
Replace matplotlib drawentityalert_graph with Bokeh version in nbdisplay.py
Changes to nbinit.py/init_notebook:
- Added documentation URLs to warnings - so it's easier to find how to fix a config problem
- added KQLMAGIC_CONFIGURATION to enable trying AzureCLI SSO by default. This allows a user to logon using az login on the Jupyter host and have KqlMagic auto-signin
- added pandas config to return schema with dataframe html to render using native nteract data browser. This allows the nteract notebook to display data using the native data browser.
b055a45@Added local function cache to GeoIP lookup to better handle repeated IPs
423e76e@Adding more queries for notebooklets - for VMComputer and DNSEvents tables
4a029b5@Add create static method to Entity class to instantiate entity from dict or pd. python Account.create(entity_properties_dict)
cebefd3@
- Added links to medium articles in ReadtheDocs
- Generated new DataQueries.rst doc
- Added KqlmagicCustom[jupyter-basic] back to core components and updating docs for this in Installing.rst
- Setting KQLMAGICEXTRASREQUIRE env var in package init.py to prevent warnings in Kqlmagic
- Reordered params in wsconfig.py WorkspaceConfig so that you can supply the workspace name as single positional parameter rather than a keyword parameter. E.g. python ws_config = WorkspaceConfig("myworkspace")
- Change kqldriver.py so that you can supply a WorkspaceConfig instance as the "connectionstr": python qry_prov.connect(WorkspaceConfig("myworkspace"))
- Change to pivot_register run function to return single "raw" result (vs. a list of one item) if it is a list of one item - this is used by notebooklets.
- Add Pivot and entities as auto-imported items to init_notebook function, so they don't need to be manually imported in the notebook.
a46b57f@Updated notebooksamples.rst with latest notebooks.notebooksamples
- Added Releases.rst - pointer to GitHub release page.

Fixes

bbc348f@
- Workarounds for some test errors in testnbinit.py and testuser_config.pyUpdating requirements-dev.txt with pip-compatible versions.
- Adding same updates to conda-reqs-dev.txt and conda-reqs-dev-pip.txt
51db655@
- Fixing test error - seems like subtle change or bug in pandas groupby behavior - in sessionize.pyChanging Development Status classifier and adding some extra keywords in setup.py
- Adding more verbose output to try to catch spurious errors in test_nbinit.py
7d7b06e@Adding beautifulsoup4 and markdown to dev/test requirements in requirements-dev.txt and conda-reqs-dev.txtAdding time unit control to QueryTime widget to allow interactive setting of day/week/month. Also increased the max range for these units - in nbwidgets.py
21cc5c4@black formatting of security_alert
cc8118c@Adding markdown to requirements-dev.txt
20db6cb@
- Corrected pip extras syntax in Installing.rst
- Invalid return type in pkgconfig.py:: validateconfig
b055a45@
- Adding file lock to unittestlib::custommpconfig - because settings are global, multiple test processes can change the settings on each other.
- Minor updates to test_nbinit.py
- Fix in pivotregister.py - _iteratefunc did not pass through **kwargs to function. Also added a few comments to explain what's going on
- Bug in securityalertgraph - if NTDomain attribute is None
- Missing update to allips perf optimizatio nin converttoipentities
- Adding filelock to dev requirements-dev.txt
229ee09@Fixing infinite recursion issue in processtreeutils.py Issue #148
98108cc@Merge remote-tracking branch 'origin/master' into ianhelle/rel1.0.0-cleanup-2021-03-12
423e76e@
- Fixing wording in SelectAlert widget
- Fixing potential None value error in GetText and GetEnvironmentText
- Fixing a few bugs and tidying in converttoipentities in iputils.py
- Random linting errors.
- Fixing issue #146 - Error is thrown when AzureSentinel config is not present in msticpyconfig.yaml file
- Added extra tests for nbinit.py in test_nbinit.py
707a201@Minor fixes to Azure Sentinel API and Entities (#141)* f-strings and entity update
- CloudError response updates
- Fix - security_alert when props are not defined
- Changed key accessor to get() calls with defaults in security_alert (so does not break if key is missing).
4a029b5@Fix bug in ip_utils.py convert to entities
7403477@Fix broken link to MPSettingsEditor.ipynb (#145)
dafdc44@Fix to provider name handling in user_config.py
cebefd3@
- Fixed QueryProviderDocumenter.ipynb bug
7b00a5c@Doc string error - prospector - in testmpextras.py
64ac03a@Fixing test error in test_nbinit.pySome fixes to notebooks (mainly getting rid of uneeded Seaborn refs)
a46b57f@Cleanup and bug fixes for v1.0 - dependencies.-
- Updated msticpy notebooks
- Updated setup.py, requirements.txt and conda-reqs with new/non-conflicting version
- Fix to testnbinit.py and importanalyzer.py
- Added test script for dependencies/extras
- Fix to README
- Removed random unicode char in PivotFunctions.rst

- Python
Published by ianhelle about 5 years ago

msticpy - MSTICPy extras test pre-release

For pre-release testing PyPi distribution

- Python
Published by ianhelle over 5 years ago

msticpy - Config editor, auto-load, dependencies and Sql2Kql translator

Overview

We've finally decided that we're ready to put a 1.0.0 release number on MSTICPy and move out of the Beta world.

There are three new features in this release and one major update.

Flexible dependencies with pip extras
Settings management and auto-load of components
An experimental SQL to KQL translator
Significant updates to the pivoting library.

New Features

MSTICPY Dependencies - Implementing Extras #128

We've split MSTICPy dependencies into "extras". This means that the base install of MSTICPy does not install every dependency. Previously, many rarely-used dependencies were installed regardless of whether they were going to be used or not. This is couple with a refactoring of the code so that, should you try to use functionality that does not have a dependency installed, an informative exception message is displayed telling you which extra you need to install.

Implemented pip "extras" for msticpy install - drastically reduces install time for core msticpy.
Refactored many modules to emit informative warning if user tries to load functionality that requires a different "extra"
Refactored unit tests to work with missing extras.
Added pyperclip to pkg dependencies exceptions.
Added extras documentation to Installing documentation
Added requirements-all.txt - that will always install all dependencies.
Added pre-commit hook to generate requirements-all.txt

MSTICPY config settings management #136

The MSTICPy configuration file has grown to be quite complex. We've tried to address this by creating some interactive tools to let you create and edit settings using a simple GUI and creating a notebook that walks you through creating your settings file for the first time.

MSTICPy itself has a number of initialization/loading steps that you need to carry out before starting to use it in a notebook. The other part of this feature is the ability to specify, in settings, what to load at initialization. Components that can be auto-loaded include: DataProviders, TI Providers, Notebooklets and Pivot functions. These are specified in the msticpyconfig.yaml (you can edit these settings with the Settings editor) and auto-loaded when you run init_notebook() at the start of your notebook.

Flexible UI for configuring MSTICPy settings
User environment configuration for notebooks - lets you specify (in settings), which providers/modules, etc. that should be loaded automatically.
Added minimal output from nbinit to show imported modules
Added checkversion in initnotebook function to indicate if a new version of MSTICPy has been released.
Added a function to retrieve and show current KV secrets
Updated MSTICPy Configuration documentation
Added MSTICPy Settings Editor documenation
Added MSTICPy Settings notebook
Added documentation diagram Config editor, auto-load, dependencies and Sql2Kql translator

SQL to KQL Translator #132

Core feature implemented as msticpy.data.sqltokql using mozsqlparser
Support for limited SparkSQL extensions
SQL TO KQL Conversion Documentation
SQLToKql Notebook

Updated Features

Pivot Functions #131

The pivot functionality has been updated with several usability and documentation improvements. Notable features include:

Pivot browser, letting you browse/search for pivot functions.
Persistent Pipelines - you can save pipelined pivot operations in a YAML file and run them on input DataFrames

Full list of changes

Made AzureSentinel and MDE the preferred names for LogAnalytics and MDE drivers.
Added pivotbrowser UI - pivotbrowser.py
Added ability to read pipeline definitions from yaml files - pivot_pipeline.py
Adding pivot.teeexec pipeline function - in pivotpd_accessor.py
Add ability to add arbitrary/ad hoc functions as pivots - in pivot.py
Exposing get_timespan function in Pivot class as public function - in pivot.py.
Added DNS entity to several pivot functions - mppivotreg.yaml
Fixed some queries for more consistency.
Pivot data query functions now prefixed with table name.
Added ability for pivot functions to return raw output.
Add joins for pivot data queries in pivotdataqueries.py
Add "print" query debug parameter in data_providers.py
Add findentity function in entities _init__.py
Add "pivots" attribute (an alias for getpivotlist) in entity.py
Add ability to set timespan more flexibly. Calling set_timespan() no longer resets the timespan.
Add PivotBrowser method to Pivot class - in pivot.py
Switched engine to "Python" for pd.readcsv in pivotmagic_core.py to handle more formatting types.
Add positional parameters to pipeline step and cleaned up code in pivot_pipeline.py
Updated PivotFunctions documentation
Updated PivotFunctions Notebook
Added PivotFunctions-Introduction notebook

Miscellaneous Updates and Additions

added SplunkProvider documentation
Updated README and package summary section of MSTICPy to align with current features

Fixes

Updated formatting for new black version
Remove unneeded code from keyvault_client.py
Fixed pivotregisterreader to skip classes that cannot be instantiated (e.g. IPStack if user doesn't have API key)
Additions/corrections to Installing.rst
Correction to FoliumMap.ipynb - removing dropna from read_csv in FoliumMap notebook
Adding vt, vt_graph to Sphinx mock list
Fixed some problems and renamed module locations in notebooks and RST docs.
Some corrections to documentation in AzureSentinel and DataAcquisition docs.
some fixes to tests for testpkgimports and import_analyzer.py
fix to config2kv.py to correct some problems
fix for ipwidgets warning about deprecated on_submit() method
multiple fixes for typos and duplicate section names in: DataProviders.rst, UploadData.rst, PivotFunctions.rst
fixed issue in nbinit.py where extra_imports were being lost.
fix for QueryTime in nbwidgets.py - exception if user types invalid value into date field.
fixed several issues in testmprelease.cmd with messed up folders/current folder.
Bandit warning on use of random.randint()
Removing test "secret" from MPSettingsEditor.ipynb triggering credscan warning

- Python
Published by ianhelle over 5 years ago

msticpy - Config editor, auto-load, dependencies and Sql2Kql translator - moved to v1.0.0.pre2

Test pre-release

- Python
Published by ianhelle over 5 years ago

msticpy - MSTICPy Pivot functions

The big feature update for this release is pivot functions. These link much of the functionality of MSTICPy to entities (like IP Address, Host, URL). This includes queries, TI and GeoIP lookups and a number of other utility functions. It means that, if you have a function that you want to run (e.g. a WhoIs lookup) you don't need to remember which module to install or what the function syntax is - you can find this function as an attribute of the IpAddress entity class. The second advantage that pivot functions bring is consistency. All the functions have a wrapper layer that normalizes both input and output. All functions can take input as a string, a list (or other iterable) or a pandas Dataframe. They all return the results as a Dataframe as well, making it easier to display/read the output and use the output as input to further functions.

New Features #127

Pivot functions
Refactored entities into separate modules and added graphing extensions
Update MordorData.rst doc with better intro section
Added function to AzureResource entity to break up resource string and expose individual elements
Adding hashaccount as separate item type to dataobfus.py
Making hash_ip more flexible - ignoring things like localhost

Fixes

Fixing the credscan suppression for testsplunkuploader
Correction to FoliumMap.ipynb
Added flexible import to deal with azure-mgmt-monitor version changes.
Bug fix and nasty workaround for old test setup removed in pkg_config.py
PR updates adding comments, some grammar fixes and obfuscation of names.
Fixed a bug in path construction for download file.
7b0b0d5@Mypy warning in new release of mypy. (#134)
Fixing lint/formatting errors in vtlookupv3.

- Python
Published by ianhelle over 5 years ago

msticpy - Azure Sentinel APIs

New Features/Updates

Azure Sentinel API Support (#122) including:
- Retrieve Security Incidents, Queries
- New authentication options to chain Kqlmagic auth from Azure auth
Improved vtlookup3 performance improvements (#117)
- Add get_object function and some date formatting to vtlookupv3.py
- Adding vtlookupv3 module to API docs.
Data obfuscation updates (#123)
- Adding hashaccount as separate item type to dataobfus.py
- Making hash_ip more flexible - ignoring masking of localhost and private IPs
- Updating documentation, tests and mapping file.

Fixes

2f2dcc2@Updating for new version of Azure monitor (2.0.0) (#133)
47be893@Bump azure-identity from 1.4.0 to 1.5.0 (#129)
efe2618@Some miscellaneous fixes to Mordor driver (#115)
413d289@Pinning pip version to 20.2.4 (temporary move to avoid build break due to new dependency behavior)
fcf5126@Adding credscan suppressions and changing a few dummy passwords to prepare for BFG git history re-write (#126)
0c557d5@corrected timeline legend order (#119) Authored-by: Noé Pion
1a04005@Adding html5lib to requirements

- Python
Published by ianhelle over 5 years ago

msticpy - Fixes to dependencies, documentation build and Mordor

Hotfix release

Fixes

08b048d@If AzureCLI section is not in msticpconfig.yaml, no longer throws exception
Updated AzureSentinel API notebook so that it has data (obfuscated).
3384c87@Updated .pre-commit-config.yaml to exclude tests from pylint and flake8 checks 7b276a3@Adding azure-mgmt-core>=1.2.1 to requirements.txt
Added exceptions to testpkgimports.py to skip vt and vtgraphapi (now extras)
d3ad345@ Fixing to mordor_driver for Mitre web site change and making web scraping from Mitre more resilient
Adding two scripts for testing pre-release msticpy.
Some additions to VTLookupV3.ipynb text and trapping for import of nest_asyncio without installing
ReadTheDocs conf.py now retrieves version number from main package build
d8cc378@Updated ReadtheDocs requirements.txt
db6e6f9@Changes to requirements,txt to remove unneeded dependencies (#114)*
Added exclusions to conf.py and setup.py to skip install of some dependencies for ReadTheDocs (e.g. Kqlmagic). These packages will be mocked in their build - previously they caused dependency conflict errors in pip install
Bug in azureauthcore.py - raise from should have been an exception object, not a class
Bug in secret_settings.py - potential uninitialized variable
Updated testmordordriver.py to remove files on completion, Also marked file download tests as CI-only
Added pylint and flake8 as pre-commit hooks - should stop fewer errors getting into build
Some tidying of code (Sourcery) in azureauthcore and secret_settings.
Updated version to 0.8.7
Fixing some typos and linting warnings in the documentation. Clarifying language.
3c87222@Updated vt-py version in requirements.txt (#113)

- Python
Published by ianhelle over 5 years ago

msticpy - AzureML Hotfix

Hotfix release for compatibility problem with Azure ML notebooks.

- Python
Published by ianhelle over 5 years ago

msticpy - Authentication, Mordor and VirusTotal v3 API

This release includes several new features: - Support for VirusTotal v3 API - Streamlined authentication for Azure using chained authentication - Azure Sentinel API support - Mordor data provider and data browser

New Features

VirusTotal V3 API support (#97 and #106) The VirusTotal team contributed a new module that gives access to the latest version of their API. This includes building of relationships between malicious items and viewing the relationship graph. (thanks to Andres Ramirez/@aramirezmartin and Juan Infantes at VirusTotal!). See the VirusTotal V3 notebook To use this feature you will need to install msticpy with the "vt3" extra pip install msticpy[vt3]
Azure Authentication (#109) We previously has several mechanism to authenticate to Azure components - mainly Key Vault and the Azure Data module. This caused unnecessary authentications. These now use a single mechanism and can also piggy-back on existing AzureCLI and Managed Service Identity (MSI) credentials. For example, if you have run az login from Azure CLI before launching your notebook the credentials from Azure CLI will be used to fetch the required authentication token for other Azure modules in MSTICPy. Note: this does not yet include Kqlmagic but we hope to have that in the next release. To see this working, check out the AzureSentinel API notebook link below.
Azure Sentinel API Support (#109) This release includes initial support for Azure Sentinel APIs to retrieve Azure Sentinel Workspaces, Alert Rules, Hunting Queries and Hunting bookmarks (more to follow in a later release). See the Azure Sentinel API notebook
Mordor data (#108) Mordor is an open source data repository of logs illustrating different kinds of adversary behavior and referenced to the Mitre ATT&CK framework. The MSTICPy Mordor package includes a notebook browser to search through and display the data sets and a dataprovider allowing you to download datasets into pandas DataFrames to use in your notebooks. See the Mordor Data notebook

Experimental Features

Split queries by time period (#110) This allows you to split a very large or long-running query into subsets split along by time interval (e.g. 1day, 1hour, etc.) The DataProvider data queries now accept a split_query_by option to split queries by time period. The time period follows the pandas Timedelta syntax e.g. "1D", "6H". The query is divided, each piece run sequentially and the results re-assembled into a single DataFrame.

Fixes

1bce3f3@Bug fix and nasty workaround for old test setup removed in pkg_config.py 0a21b59@Updating pre-commit version for black 36d2539@Updated formatting for new black version 7897cb9@Moving VT3 dependencies to extras. (#111)

- Python
Published by ianhelle over 5 years ago

msticpy - GetEnvironmentString-hotfix

Hot fix for - GetEnvironmentString widget throws exception when variable is already defined

- Python
Published by ianhelle over 5 years ago

msticpy - Azure-mgmt-monitor fix

This is a maintenance release to temporarily fix an import conflict with the release of 1.0.0 of azure-mgmt-monitor.

It also includes a pre-release version of the VirusTotal V3 API library

- Python
Published by ianhelle over 5 years ago

msticpy - New Widgets, Data Obfuscation, TimeSeries time range extraction

New Features

#90 - new gettext and option button widgets.
Registered widgets - some widgets (GetText and QueryTimes) now have a "memory" - so if used in Jupyter notebooks and you accidentally re-run the cell they will revert to their last settings rather than the defaults.
Registered widgets can also accept notebook parameters from papermill
Added query_browser module
Added method to tilookup and data_providers modules to access respective browser widgets.
#93 Added class diagrams for DataProviders, TIProviders and Secrets sub-packages.
#94 - Added function to extract anomalous time ranges from time series anomaly results
Added functions to obfuscate/hash data to protect PII
#96 kql query failure and nbinit notebook initialization now have friendly errors
Updated data queries list

Fixes

#90 Splunk and data_providers fixes
Tooltip formatting fix for datetimes in timeline and timeseries Bokeh tools
#92 - Fixes for deprecated networkx API
#93 Error in pkg_config validate when no config sections are populated.
#95 - Removed redundant typing library from requirements
#96 - Miscellaneous linter and spelling fixes
Error in pkg_config validate when no config sections are populated.
Updated Splunk queries to use datetime type for parameters.
Adding some extra checks for null entries in msticpyconfig in pkg_config
Making nbinit skip but report any exceptions while validating msticpyconfig
Change dataproviders to use custom paths outside of the package
Fix to entity entityschema
Fix to kql_driver to handle running if not in IPython.
Add version param to test-pypi-test-pkg.cmd help.
Unit test for query_browser.
Formatting using updated black v20
Fixing conda-reqs-pip.txt requirements
Updating msal req to ~=1.0.0
Fixing mypy errors, incorrect annotation in query_source
Fixing time format bug in timeline
3fc07e7@timeformat re-ordering
9d20842@updated requirements for dateutil bug-fixes

- Python
Published by ianhelle over 5 years ago

msticpy - Dependency version fix

Maintenance release to fix import errors caused by incorrect package dependency versions. Also includes minor documentation updates.

- Python
Published by petebryan almost 6 years ago

msticpy - Splunk data connector and data uploaders

The main features of this release are a data provider for Splunk and data uploaders for Azure Sentinel and Splunk. The Splunk provider uses the Spunk SDK and makes it eas to use msticpy functionality with results from Splunk queries.

The data uploaders let you upload results or additional data to either Azure Sentinel or Splunk.

New Features

splunk connector (#81)

Authentication and initialization unified with existing data providers
Parameterized template queries runnable from data provider class
Retrieve and display saved searches as part of the available query set
Splunk provider documentation notebook

Data Uploaders (#87)

uploaders from pandas DataFrame, CSV/delimited file and folder
upload to Azure Sentinel or Splunk
documentation notebook and read-the-docs page

Data providers (#81)

fc046a5Added support for populating dynamic query set at connect time (after connect)
Added populating queries with SavedSearches in splunk_driver
Added provider specific formatting for parameters (e.g. how a given query language expects a list of items or a datetime to be formatted)
Support for fully hierarchical query list (to help organize queries into categories, subcategories, etc.)

4bbf785 Blackhat Demo Notebook

Fixes

87dab39 Adding unit tests for kql and splunk drivers (latter has a failure) 7123511 Bug in dataproviders and paramextractor e2ea5c6 Fix for tooltip formatting for timeline charts 2a5a734 Error in pkg_config validate when no config sections are populated. a141f3b Temporarily restricting pandas version to <=1.0.5 9901b72 TI Browser widget d1e6430 Fixes to splunk driver test

- Python
Published by ianhelle almost 6 years ago

msticpy - TI Results Browser

New Features

Threat Intelligence results browser widget (#84)

Fixes

805d71d@ Error in pkgconfig validate when no config sections are populated. (#85) 860f7a4@Fixing URLs in exception help links f92ef4d@update SelectAlert 1a72495@Fixing mypy error in wsconfig Fixing odd test break in tiproviders (didn't have default msticpyconfig in current directory) Removing redundant test in testnbtools Fixing bug in unittestlib - setting ENV VAR to "" instead of removing it. a031c2a@Incorrect capitalization in Readme c6468aa@Left in an unwanted cell from testing d8098ee@Test error in EventClustering notebook 1c14a12@Fixing timestamp Timezone issue in processtreeutils and eventcluster a08a348@Added User exception to BrowshotAdded exception handling to iputils.defwhoisinfo Fixed Output widgets where action function returns single dataframe Fixed syslogutils.clustersysloglogonsdf so that it uses only source data to generate datetime objects - this should avoid TypeErrors from pandas when comparing datetimes where one dt has TZ info and the other doesn't. As long as the source is consistent this should avoid this error. 4f73b44@Making pkginstall and notebook init errors more friendly 1dafc1f@Fixed problem with test cases not being able to do absolute imports of msticpy modules(deleting rogue _init.py at root of package Updating Readme to make images clearer. Add checkversion to root _init.py to check version against latest on PyPI Also added check version code to tools. Minor change to nbinit.checkconfig() to make testing easier. 8bbf417@Prevent default output from queries (Kqlmagic post 0.1.111) 8eff722@Fix for the problem with pandas datetime representation - you cannot compare timezone-aware and timezone-naiive dates. Redoing logic so that the code only uses timezone awareness state from data values. This should work as long as all or none of the data sets have timezone info.

- Python
Published by ianhelle almost 6 years ago

msticpy - Package Conflict Fix

Maintenance release to fix some package version conflicts.

- Python
Published by ianhelle almost 6 years ago

msticpy - Friendly Exceptions

Mostly housekeeping release. We've introduced a set of friendly exceptions for common configuration-related problems that users might encounter while using msticpy in notebooks. In most cases, the cause is lack of a config item such as an API key or setting. Hitting one of these exceptions gives you a few lines of explanatory text plus links to documentation on readthedocs to help you fix it. In most cases the traceback is irrelevant so this is not shown for these exception types (although this can be re-enabled).

Adding some new capabilities like being able to use our time series analysis module on any time-stamped log data. We've also done a lot of documentation additions/improvements - for Anomalous Sequence, TimeSeries and msticpy configuration plus miscellaneous freshening and improving.

A lot of the other work is related to compatibility with nteract environment in Azure Machine Learning - this will soon be the default notebook environment for Azure Sentinel.

New Features

Friendly exceptions and exception framework for notebooks #73
Replaced the use of the ipywidgets Output widget with IPython updatable display() objects. nteract/AML compat. #71
Added timeseries decomposition (using Statsmodel STL) so that you can do time series analysis on any data #69 (previously we relied on Kql/Azure Sentinel to do the decomposition/analysis part). Documentation and notebook added
Update to Anomalous Sequence modules
- Added modellable_params argument in model classes - this can be used to override the default of using rough heuristics to decide which params have modellable values, and instead, manually specify the parameters. #65
- Added sample notebook and RST/ReadtheDocs documentation for Anomalous Sequence.
Added checkversion() function that will check the current version against latest on PyPI ``` import msticpy msticpy.checkversion() ```
Relaxing some version requirements in setup.py/requirements.txt to match AzureNotebooks resulting in quicker install #68
Updated display of logon details in nbdisplay
Updated README to add missing details, correct some things and add a few more images.
Added documentation for Azure Sentinel configuration for notebooks - how to use the various config files.
Updated a lot of the introductory sections to readthedocs to bring up-to-date and align with README
Replace some badly-formatted tables in readthedocs docs
Added analysis subpackage to docs so that Anomaly Sequence and TimeSeries module/API docs appear in the API doc tree (readthedocs)
Adding new queries for Notebooklets project #67

Fixes

Some fixes to comp_reqs.py test tool to show missing packages and handle version comparison operators.
New release of pandas/numpy surfaced a bug where we were doing datetime comparisons between timezone naive and timezone-aware datetimes - fixed in timeline, processtree and eventcluster.
New behavior in Kqlmagic (> 0.1.111) causes queries in the middle of a cell to output by default. Added workaround to suppress this.
Rogue init.py in root of repo (had been there forever) was preventing test cases using absolute imports of tested modules. removed the offending file and updated all tests to use absolute imports
Seems that we'd been inadvertently (at some point) including test files in our setuptools/PyPI package - these should now be gone.
Fixed an issue with GeoIPLite (maxmind) database download and cleaned up logic.
Added better exception handling in nbinit
Making pkginstall and notebook init errors more friendly
Removing some deprecated terms.
Fixing timestamp Timezone issue in processtreeutils, eventcluster, processtree and syslogutils.clustersysloglogons_df
Fixing/constraining some azure dependencies for Sphinx #68
Fix for schema property in kql_driver #70

- Python
Published by ianhelle almost 6 years ago

msticpy - LocalData Provider

New Features

db86480:
- LocalDataDriver for using CSV and pickled DF files as a QueryProvider (#64) This is primary for demonstration and test purposes where you do not have access to online data sources. It replicates the functionality of QueryProvider allowing drop-in replacement in existing notebooks.
- Updated DataQueries.rst ReadtheDocs page with new queries
- Add documentation for LocalDataDriver to DataProviders.rst and updated section on creating query files.
66a66d2:
- Checked in notebook to create DataQueries.rst
- Added "AzureSentinel" alias for LogAnalytics DataEnvironment

Breaking Changes

db86480:
- Removed deprecated kql.py, querybuiltinqueries, querymgr.py, queryschema.py
- Changed location of query_defns.py and made pkg reference updates in several modules and notebooks.
- Some fixes to support localdatadriver in querystore.py, driverbase.py and data_providers.py
- Unit test - testlocaldataqueries.yaml and supporting data and query files.
- Fixed test in test_utils.py to work on Linux
- Reduced warnings produced during pytest run to something more reasonable (mainly by removing deprecated code
8a32ad5:
- Changed tilookup and kqlbase/kqldriver so that handling failure to load is a bit friendlier. E.g. running TILookup in a non-IPython environment (with ASTI provider) will now just cause a warning, not an exception.
- kqldriver.py also updated to check for getipython() returning None and output friendlier message.
- Changed driver_base.py and derived class to take additional QuerySource parameter for query() method - not yet used but required so that we can implement driver-specific checks on query parameters.

- Python
Published by ianhelle about 6 years ago

msticpy - Anomaly Sequence Detection and Morph Charts (re-release)

This release includes:

Anomaly sequence analysis and visualization using Markov chain @karishma-dixit Morph Chart visualization of log events @petebryan

(originally released as v0.4.1 but updated to v0.5.0)

New Features

Anomalous sequences (#60) Markov Chain anomaly analysis for sequences of commands/patterns in a session
Morph Charts visualization - 3D visualization of event data using experimental (#58) Morph Charts exploration
nbinit: a neater and more robust startup/setup function for Jupyter notebooks handling package installs, imports and option setting (#62)
Azure Sentinel Queries
- Added two Logon fail queries for Linux (#62)
- Add Linux logons for host
Added msticpy.common.pkgconfig.validateconfig() to validate current config or external config file (#62)

Fixes

f78a29e:
- Change return type on for bokeh graphs to return whole layout
- Improved geoip error messages when Api key is missing
- Fixing bug in pkg_config if no workspaces are defined (empty workspaces key)
31cb17f: Added context manager to temporarily set msticpyconfig to another path and auto-revert settings afterwards.
827477b: make titles consistent on the widgets page (#59)
7964b5f: Fix to utility.py - checkandinstallmissingpackages to all package version to be specified.
f793d55:
- Updated pkg_config to allow AzureCLI and AzureSentinel sections to use Key Vault protection of the keys and use of Env Vars, etc.
- Timeline - fixed Tooltip representation of Timestamps for different representations of numpy's types
- Fixed an error in test-pypi-test-pkg.cmd
3e42e42: Doc fix and OutOfBoundsDatetime catch
efc3d69: OTX TI Provider fixes to encode URL IoC prior to submitting (#55)
0ad166a: fixing headings in rst docs for timeseries
606fc8f: Fixing broken Readthedocs link (#53)
4810e1f: Fixing some documentation omissions/errors (#52)
43bbd3c: Updating pylintrc to change limits for some checks.
f50eec2: Notebooklet queries and timeline hide option
13c3f3f Flake8 error with unknown "QuerySource" (#63)
9921352 Adding pkgs to conda-reqs-pip.txt Removing Python 3.7 version setting from pre-commit
921370c (#63)
- requirements.txt and setup.py changes to avoid version conflicts (causing sphinx to fail)updated version to 5.0
c900386 Fixed issue causing test failure (#63)
5c9db2d Adding getallentities feature used in Alerts Notebook (#63)

- Python
Published by ianhelle about 6 years ago

msticpy - Anomaly Sequence Detection and Morph Charts

This release includes:

Anomaly sequence analysis and visualization using Markov chain @karishma-dixit Morph Chart visualization of log events @petebryan

New Features

Anomalous sequences Markov Chain anomaly analysis for sequences of commands/patterns in a session
Morph Charts visualization - 3D visualization of event data using experimental Morph Charts exploration
nbinit: a neater and more robust startup/setup function for Jupyter notebooks handling package installs, imports and option setting
Azure Sentinel Queries
- Added two Logon fail queries for linux
- Add Linux logons for host
Added msticpy.common.pkgconfig.validateconfig() to validate current config or external config file

Fixes

f78a29e:
- Change return type on for bokeh graphs to return whole layout
- Improved geoip error messages when Api key is missing
- Fixing bug in pkg_config if no workspaces are defined (empty workspaces key)
31cb17f: Added context manager to temporarily set msticpyconfig to another path and auto-revert settings afterwards.
827477b: make titles consistent on the widgets page (#59)
7964b5f: Fix to utility.py - checkandinstallmissingpackages to all package version to be specified.
f793d55:
- Updated pkg_config to allow AzureCLI and AzureSentinel sections to use Key Vault protection of the keys and use of Env Vars, etc.
- Timeline - fixed Tooltip representation of Timestamps for different representations of numpy's types
- Fixed an error in test-pypi-test-pkg.cmd
3e42e42: Doc fix and OutOfBoundsDatetime catch
efc3d69: OTX TI Provider fixes to encode URL IoC prior to submitting
0ad166a: fixing headings in rst docs for timeseries
606fc8f: Fixing broken Readthedocs link (#53)
4810e1f: Fixing some documentation omissions/errors (#52)
43bbd3c: Updating pylintrc to change limits for some checks.
f50eec2: Notebooklet queries and timeline hide option

- Python
Published by ianhelle about 6 years ago

msticpy - Azure Data, TimeSeries and Key Vault Secrets

This release includes:

Expansion of Azure Data API for retrieving additional data about subscriptions and resources from Azure APIs.
Time Series anomaly detection for arbitrary Kusto data sets together with visualization of time series charts in Jupyter Notebooks using Bokeh Charts.
Using KeyVault and Python Keyring to store secrets used to authenticate to web data providers. Examples include API keys for Threat Intel and Geo IP Providers. Other provider types will be included in a future release.

New Features

Azure data expansion and documentation
Keyvault and keyring secrets management with support for multiple Azure clouds
- config2kv.py KV secret update tool
Timeseries - Bokeh with KQL and documentation
- KQL generic time series decomposition queries
- Bokeh time series visualization
Added pandas version of getwhoisinfo and added as DataFrame accessor function.
Added cmd script to test PyPi test deployment
Added Conda package requirements files
Updated TI providers to provide more consistent output and reduce false positives
- Using text rather than number to express severity
Made TISeverity class comparable and parsable from string or int
Added mpdemodata.py notebook helper to tools.
SecurityAlert has more flexible recognition of entities
Added additional dependencies for azure mgmt, keyvault and others.

Fixes

Fixed getiptype ordering to return more accurate IP types
Fix entity extraction in SecurityAlert to allow nested entities to work correctly
Additional test cases

- Python
Published by ianhelle about 6 years ago

msticpy - Pandas Extensions

This release includes early implementations of pandas extensions so that you can invoke msticpy functionality directly from a DataFrame: python my_events_df.mp_timeline.plot() my_proc_events_df.mp_process_tree.plot() So far, IoCExtract, Base64Unpack, Timeline and ProcessTree have these extensions but we will be adding them to more msticpy modules over time.

Also, IoCExtract and Base64 decode functions have IPython magics (%%ioc and %%b64) allowing you to paste a block of text into a notebook cell and run the function directly on that text.

Most other changes are primarily maintenance and house-keeping improvements such as increasing unit testing code coverage.

New Features

process_tree - added pandas extension and changed main function so that it returns the plot figure and layout
timeline - added pandas extension. added support for DateTime column in Tooltips (display as date time rather than number)
base64unpack - added pandas extension, added IPython %%b64 magic
iocextract - added pandas extension, added IPython %%ioc magic
Added documentation and notebook examples for the pandas extensions and magics.
wsconfig - added method to display available workspaces
README.MD - added some graphics to brighten the page up a little
Added unit test test_folium.py
Adding FoliumMap.ipynb sample notebook
Added additional geolocation centering functions for FoliumMap
Updates to GeoIPLookups.ipynb
Add parameter checks to timeline.py and process_tree.py so that invalid **kwargs produced a helpful error message.
Added requirements-dev.txt

Fixes

Typos in AzureData.rst
Adding GeoIP tests.
Removing deprecated lines from coverage reports.
Cleaned up pytest coverage report.
Adding suppression file for credscan false positives
Removing SecurityAlertandEntities notebook with misleading content
Removed failing cell from end of GeoIPLookups notebook
Fixed a few errors in foliummap.py
Fixed bug in GeoIP DB downloader
Changed foliummap center functions to use median by default
Removed largely redundant os_family param from iocextract.py functions
Fixed sectools_magics iocextract class
Update testiocextractor for new parameters
domaintools - changed tldindex and ssl_bl attributes to properties that auto-load on first use (prevents remote http request if data on class instantiation)
Added more tests for utility.py
Add environment variable to selectively run some long-duration tests during build only (these are no skipped in local tests)
Tidied up/refactored some code in base64unpack.py

- Python
Published by ianhelle over 6 years ago

msticpy - mstipy User guide

New Features

The documentation now includes a user guide covering many aspects of msticpy It includes the following sections: - Getting started section (Installation and configuration) - Data Acquisition (querying and data) - Data Enrichment (GeoIP, Threat Intel) - Data Analysis (IoC extraction, decoding, clustering - Visualization (Event timeline, Process tree, Mapping, widgets)

Documentation is on ReadTheDocs

Fixes

Broken links and outdated docs updated
Fixes to some unit tests

- Python
Published by ianhelle over 6 years ago

msticpy - MDATP, Azure, ProcessTree

Overview

Note: This release is consolidation of v0.2.8 and some additional features and fixes. Adding everything to the release notes for this version since the previous one did not get published to PyPi.

This release contains three important features: - Query support for Microsoft Defender ATP - Interactive Process Tree browser. - Support for querying Azure properties for subscriptions and resources

New Features

Microsoft Defender Query Support. Added a query provider/driver to query Defender alerts, machines, processes and arbitrary KQL queries of the Hunting data
Template queries for MDATP for hunting and standard entities
Process Tree Viewer - Bokeh interactive graphical view for one or more Process Trees in a data set. Supports both Windows and Linux.
Process tree utilities - data library to create and query process trees.
Azure properties of subscriptions and resources such as VMs can be queried from Notebooks.
Query providers now accept ISO-string format for datetime fields for queries (in addition to datetime and timedelta)
Added Progress widget to nbwidgets.
Added config support for GeoIP providers from msticpyconfig.yaml
- GeoIP classes try to obtain API key from config if not supplied
Refactored tiprovidersettings to generic provider_settings module

Fixes

Miscellaneous linting/checker bugs
Spelling and path errors in docs
Fixing paths for https://github/Azure/Azure-Sentinel-Notebooks repo.
Updating dependency to Bokeh 1.40
Fixed timeline legend bug
Fix for Maxmind requiring authentication for GeoLite DB download (GeoIP classes are no longer loaded by default.
Added missing pytz and pyyaml packages to requirements.txt and setup.py.
GeoLite2 url and archive extraction changes

- Python
Published by ianhelle over 6 years ago

msticpy - MDATP/AzureData Support and ProcessTree Viewer

Overview

This release contains three important features: - Query support for Microsoft Defender ATP - Interactive Process Tree browser. - Support for querying Azure properties for subscriptions and resources

New Features

Microsoft Defender Query Support. Added a query provider/driver to query Defender alerts, machines, processes and arbitrary KQL queries of the Hunting data
Template queries for MDATP for hunting and standard entities
Process Tree Viewer - Bokeh interactive graphical view for one or more Process Trees in a data set. Supports both Windows and Linux.
Process tree utilities - data library to create and query process trees.
Azure properties of subscriptions and resources such as VMs can be queried from Notebooks.
Query providers now accept ISO-string format for datatime fields for queries (in addition to datetime and timedelta)
Added Progress widget to nbwidgets.

Fixes

Miscellaneous linting/checker bugs
Spelling and path errors in docs
Fixing paths for https://github/Azure/Azure-Sentinel-Notebooks repo.
Updating dependency to Bokeh 1.40
Fixed timeline legend bug

- Python
Published by ianhelle over 6 years ago

msticpy - November 2019

Overview

Most of the features and fixes this month were done to support the new Entity Explorer series of notebooks published on Azure-Sentinel-Notebooks repo

New Features

Added syslog_utils module for common syslog data manipulation
Added cmd_line module for syslog CMD analysis
Added ip_utils module for common IP Address operations
Added domain_utils module for common domain/DNS/URL operations
Added new TI providers - openpagerank and torexitnodes
Added package auto-install function added to utility.py
nbwidgets: added filtering to all select/list widgets
In query templates you can now express date parameters (and query defaults) as a pos/neg number relative to current date or a KQL-like range (-30d, 15h, etc.)
wsconfig.py - can create a Connection string directly from wsconfig.
Added settings as a top-level msticpy attribute.
TI dataframes derive their column names directly from LookupResult so don't need to keep two lists.
A number of new queries were added in the following categories:
- Syslog/linux
- Office Activity
- Azure/Azure Active Directory
- Azure Network Analytics
- KQL time series queries
- Hunting bookmarks
Documentation updates:
- new documentation page on msticpy configuration - msticpyconfig.yaml
- Updated Readme.md for new features.

Fixes

Typo in wsconfig.py - wsconfig throws meaningful error if config values are not found
fix yaml parsing error in timeseries kql
Miscellaneous fixes from notebook testing for new Explorer notebooks
Changed param_extractor to always prefer supplied params over defaults
tilookup fix - exception thrown if an empty IoCs list sent to it
geoip - fixed multiple problems with the DF lookup version of the API
Updated Pandas requirement to 0.25
Query fixes - miscellaneous
Added dependencies on cryptography, tqdm, ipwhois

- Python
Published by ianhelle over 6 years ago

msticpy - Multi-timeline Interactive Visualization

New Features

Major update to display_timeline control:
- allows arbitrary number of event series
- input as dict of data series or grouped DataFrame
- added interactive visual time range control
- added displaytimelinevalues to display timelines with a scalar value (line, circle, vbar)
- added sample notebook
- added ReadtheDocs page describing usage
- Moved to timeline.py module

Other Improvements/Additions

Updated Base64Unpack, EventClustering, NotebookWidgets and TIProviders notebooks.
Added unit test capability for UI-dependent packages by running notebooks within the unit test
nbwidgets: added filtering text box to all select widgets
nbwidgets: added SelectSubset widget allowing you to pick from one list and add to selected subset
Updates to documentation/README.md
Added checks for no TI Providers or missing keys and updated TIProvider docs for this.
Added network data query yaml - kqlsentwinevent.yaml
Added WinSecurityEvent.json events file
Added pre-commit hooks including local hook script download_tlds.py

Fixes

Fixes from testing notebook development:
Minor change to base64unpack.py to prevent pandas warning
entityschema: fixing repr to always return a string
securitybase: removing broken and deprecated properties adding _repr__
ti_lookup - remove unneeded import
nbwidgets - bug in restoring current index in selected items list
eventcluster - first/last time range for clustered events was not properly calculated.
Fixed foliummap error to display in notebook (implemented reprhtml_ so that instances display directly in notebook.)

- Python
Published by ianhelle almost 7 years ago

msticpy - Azure Sentinel TI Provider

New Features

Azure Sentinel Threat Intel provider in TILookup
kql_base.py provider for TILookup to support other LogAnalytics TI sources
Refactored unit tests for TIProviders with mocking of data sources.
TIProviders notebook and ReadTheDocs TI Providers doc page.
Added package config and ability for WSConfig to get workspace and tenant config from msticpyconfig.yaml

Fixes

Fixes for mypy warnings - now mypy clean
Addressed most other linting warnings
Fixed broken multiple TI lookups for http providers
Black formatting
Bug in geoip that would throw exception for private IP addresses and issue warnings rather than exceptions if something goes wrong with GeoLite DB download
Fixed errors in several network query definitions
IoCExtract bug when trying to download TLD file offline

- Python
Published by ianhelle almost 7 years ago

msticpy - Enabling data library support

Some of the data library support was not included in the previous release. This is just catching up.

- Python
Published by ianhelle about 7 years ago

msticpy - Data Query Library

New Features

Data query library supporting multiple data providers (kqlmagic, Odata...).
Query definitions are stored in YAML files in data/queries. Allowing default parameters and replaceable parameters at runtime.

Fixes

Updates to IocExtract and base64unpack.

- Python
Published by pareid about 7 years ago

msticpy - Minor README/Setup Updates

New Features

Updated README with links to documentation and sample notebooks.
Added links to code and documentation in setup.py allowing links to appear in the PyPI repo side menu.

Fixes

Fixed a broken project url in setup.py.

- Python
Published by pareid about 7 years ago

msticpy - Sphinx Documentation

New Features

Adding Sphinx documentation for Read the docs Numpy docstrings should now be used (almost) everywhere Added function to kql.py execute simple kql string query. Added function to auditdextract.py to read audit logs from file. Added these HowTo docs to repo:

Jupyter And Security
Azure Sentinel and Jupyter
Enabling Auditd on Linux in Azure Sentinel

Fixes

Fixing some errors in iocextract Pylint and Flake8 warnings (mostly reducing line length to < 90)

- Python
Published by ianhelle about 7 years ago

msticpy - Adding test cases and correcting linting warnings

New Features

Added observationlist module

Fixes

Fixing linting warnings. Adding unittests testeventcluster, testobservationlist, testsecurity_event + test data Miscellaneous small fixes and improvements.

- Python
Published by ianhelle about 7 years ago

msticpy - Adding documentation notebooks

New Features

Added several doc notebooks for components Updated Readme.md Initial refactoring of docstrings to use numpy standard format. Additional unit tests.

Fixes

Fixing some bugs found while doing the documentation.

- Python
Published by ianhelle about 7 years ago

msticpy - v0.1.3

msticpy initial public release v0.1.3

- Python
Published by pareid about 7 years ago

Recent Releases of msticpy

msticpy - Compatibility release - Bokeh, VirusTotal, AzureCredentials

What's Changed

msticpy - Maintenance release QueryEditor, PrismaCloudDriver

Highlights

What's Changed

msticpy - Cyberint TI provider and Prisma Cloud (Palo Alto) Data provider

Prisma Cloud Driver

Cyberint TI Provider

Azure Sentinel/Azure Monitor

831 is specifically the problems with the experimental driver

Certificate Authentication support for OData drivers (Defender and MSGraph)

Other changes

What's Changed

New Contributors

msticpy - Multi-dimensional plots for outliers

Highlights

Multi-dimensional plots for outliers by @Tatsuya-hasegawa

create random numeric samples

calc outliers by Isolation Forest algorism

Improved code/docs for federated authentication for M365D/M356 Graph providers - @ryan-detect-dot-dev

Rigorous Type Annotation work started by @FlorianBracq earlier this year continues.

Other fixes

What's Changed

New Contributors

msticpy - User Session Management, MaxMind Geolit fix, Extract nested dicts from Pandas

User Session Configuration

Maxmind GeoIPLite fix

Extract nested dictionaries from pandas column to multiple rows/columns

What's Changed

New Contributors

msticpy - Hotfix for authentication error

What's Changed

msticpy - AI documentation assistant, BinaryEdge TI provider and other misc fixes

What's Changed

New Contributors

msticpy - Splunk and Sentinel Updates

Sentinel updates

Splunk Updates

Other fixes

What's Changed

New Contributors

msticpy - Sentinel Split Query fix

What's Changed

New Contributors

msticpy - v2.10.0

What's Changed

New Contributors

msticpy - Defender Advanced hunting, IPQualityScore TI provider

IPQualityScore

Defender Advanced Hunting API

Startup errors when running in unexpected environments.

Startup fixes and perf improvements

Azure env credentials fix

Improvements to Entities

What's Changed

New Contributors

msticpy - Stability release

What's Changed

New Contributors

msticpy - 2.8.0 pre-release

msticpy - TI Providers, Sentinel/Kusto Drivers, Query Editor

Main Changes in this release

Two new TI Providers

Updated Data providers for Sentinel/Azure Monitor/Log Analytics and Kusto/Azure Data Explorer

Query Editor

Updates to Authentication.

Other major items

Thanks

What's Changed

New Contributors

msticpy - DataProviders, QueryEditor, CrowdSec and AbuseIPDB TIProviders

Preview release of 2.7.0

Main Changes

Two new TI Providers:

Updated Data providers for Sentinel/Azure Monitor/Log Analytics and Kusto/Azure Data Explorer

Query Editor

Updates to Authentication - esp for the AzureData and MicrosoftSentinel API modules

Other items

What's Changed