[Security Solution][Sourcerer] Replace Sourcerer with Discover Data View Picker

[lgestc]

Unified Data View Picker: Phase 1 Implementation

Part of https://github.com/elastic/security-team/issues/11959

What This PR Does

This PR represents the first step in our transition from the current Sourcerer component to the new unified Data View Picker. Specifically, this implementation:

• Creates a new Data View Picker component
• Implements feature flag protection for all changes
• Handles asynchronous effects through Redux listener middleware
• Establishes a new Redux store architecture to support ad hoc data views infrastructure
• Utilizes ad hoc data views to handle legacy patterns from series 7 (replacing the previous upgrade data view flow)

See the readme for more info:
x-pack/solutions/security/plugins/security_solution/public/data_view_manager/readme.md

What This PR Does NOT Cover

• Does not affect screens other than Timelines
• Does not modify the existing Sourcerer component in any way
• Does not fully support all URL/local storage patterns

Implementation Notes

We've made several accommodations to support both Sourcerer and the new Data View Picker simultaneously during this transition period, including:

• Some interfaces might look odd, especially the hooks that return the data view or patterns - this is intentional to support existing use cases
• There are feature flag-based conditional statements throughout the code that will be removed once the transition is complete

Testing Instructions

1. Add the following feature flag to your configuration:

   xpack.securitySolution.enableExperimental: ['newDataViewPickerEnabled']
   

2. Navigate to the Timelines interface
3. Test interactions with the new Data View Picker

Todo:

• mention adhoc dataviews / their role in handling legacy timelines.
• rename dataViewPicker to dataViewManager

WORK IN PROGRESS But reviews are welcome

[michaelolo24]

can we default to an empty string if all the other parameters are provided in the reducer and not do these nullish checks everywhere the id is called?

[lgestc]

eventually we will only pass the spec reference around, I imagine. Unfortunately this is how the DataViewSpec looks like 😭

[michaelolo24]

Thanks for doing it this way. I understand it's not ideal, but it keeps us safe-guarded for the time being 👍🏾

[kqualters-elastic]

safer to do something like

const { browserFields: sourcererBrowserFields, sourcererDataView } = useSourcererDataView(SourcererScopeName.timeline);
  const { dataView: experimentalDataView } = useDataView(DataViewManagerScopeName.timeline);
  const experimentalBrowserFields = useBrowserFields(DataViewManagerScopeName.timeline);

  const finalBrowserFields = useMemo(() => 
    newDataViewPickerEnabled ? experimentalBrowserFields : sourcererBrowserFields,
    [newDataViewPickerEnabled, experimentalBrowserFields, sourcererBrowserFields]
  );

  const finalDataView = useMemo(() =>
    newDataViewPickerEnabled ? experimentalDataView : sourcererDataView,
    [newDataViewPickerEnabled, experimentalDataView, sourcererDataView]
  );

let sometimes leads to non-obvious issues, and of all the props in kibana, browserFields is the one to be most defensive about

[michaelolo24]

I would just call setDataview(experimentalDataView) - if you're trying to avoid rerenders, then I would just use useRef instead.

[michaelolo24]

same thing here re defaulting id to '' in the experimentalDataView initialization

[vitaliidm]

useFullDataView view called here and then dataView is getting replaced by experimentalDataView later only when newDataViewPickerEnabled is enabled.

It does look, that experimentalDataView is getting initiated every time, in disregard of feature flag.
Should it be avoided?

[kqualters-elastic]

agree, avoiding use of let would achieve the same thing, and is imo a good rule of thumb to follow

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 1cf669e

Failed CI Steps

• FTR Configs #29

Test Failures

• [job] [logs] FTR Configs #29 / lens app - group 5 lens drag and drop tests keyboard drag and drop should duplicate an element in a group

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	7056	7070	+14

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	8.9MB	8.9MB	+11.8KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	85.5KB	85.5KB	+28.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
securitySolution	577	579	+2

References to deprecated APIs

id	before	after	diff
securitySolution	351	355	+4

Total ESLint disabled count

id	before	after	diff
securitySolution	660	662	+2

History

• 💔 Build #281405 failed 069393d
• 💔 Build #281353 failed 33406cf