By Rajiv Shringi, Vinay Chella, Kaidan Fullerton, Oleksii Tkachuk, Joey Lynch
As Netflix continues to develop and diversify into varied sectors like Video on Demand and Gaming, the flexibility to ingest and retailer huge quantities of temporal information — usually reaching petabytes — with millisecond entry latency has grow to be more and more very important. In earlier weblog posts, we launched the Key-Value Data Abstraction Layer and the Data Gateway Platform, each of that are integral to Netflix’s information structure. The Key-Value Abstraction gives a versatile, scalable answer for storing and accessing structured key-value information, whereas the Data Gateway Platform gives important infrastructure for safeguarding, configuring, and deploying the information tier.
Building on these foundational abstractions, we developed the TimeSeries Abstraction — a flexible and scalable answer designed to effectively retailer and question massive volumes of temporal occasion information with low millisecond latencies, all in an economical method throughout varied use instances.
In this publish, we are going to delve into the structure, design rules, and real-world functions of the TimeSeries Abstraction, demonstrating the way it enhances our platform’s skill to handle temporal information at scale.
Note: Contrary to what the title might counsel, this method just isn’t constructed as a general-purpose time sequence database. We don’t use it for metrics, histograms, timers, or any such near-real time analytics use case. Those use instances are effectively served by the Netflix Atlas telemetry system. Instead, we deal with addressing the problem of storing and accessing extraordinarily high-throughput, immutable temporal occasion information in a low-latency and cost-efficient method.
At Netflix, temporal information is constantly generated and utilized, whether or not from person interactions like video-play occasions, asset impressions, or advanced micro-service community actions. Effectively managing this information at scale to extract precious insights is essential for guaranteeing optimum person experiences and system reliability.
However, storing and querying such information presents a novel set of challenges:
- High Throughput: Managing as much as 10 million writes per second whereas sustaining excessive availability.
- Efficient Querying in Large Datasets: Storing petabytes of knowledge whereas guaranteeing major key reads return outcomes inside low double-digit milliseconds, and supporting searches and aggregations throughout a number of secondary attributes.
- Global Reads and Writes: Facilitating learn and write operations from anyplace on the planet with adjustable consistency fashions.
- Tunable Configuration: Offering the flexibility to partition datasets in both a single-tenant or multi-tenant datastore, with choices to regulate varied dataset facets resembling retention and consistency.
- Handling Bursty Traffic: Managing important site visitors spikes throughout high-demand occasions, resembling new content material launches or regional failovers.
- Cost Efficiency: Reducing the associated fee per byte and per operation to optimize long-term retention whereas minimizing infrastructure bills, which might quantity to hundreds of thousands of {dollars} for Netflix.
The TimeSeries Abstraction was developed to satisfy these necessities, constructed across the following core design rules:
- Partitioned Data: Data is partitioned utilizing a novel temporal partitioning technique mixed with an occasion bucketing strategy to effectively handle bursty workloads and streamline queries.
- Flexible Storage: The service is designed to combine with varied storage backends, together with Apache Cassandra and Elasticsearch, permitting Netflix to customise storage options primarily based on particular use case necessities.
- Configurability: TimeSeries gives a variety of tunable choices for every dataset, offering the flexibleness wanted to accommodate a wide selection of use instances.
- Scalability: The structure helps each horizontal and vertical scaling, enabling the system to deal with rising throughput and information volumes as Netflix expands its person base and companies.
- Sharded Infrastructure: Leveraging the Data Gateway Platform, we will deploy single-tenant and/or multi-tenant infrastructure with the required entry and site visitors isolation.
Let’s dive into the varied facets of this abstraction.
We observe a novel occasion information mannequin that encapsulates all the information we wish to seize for occasions, whereas permitting us to question them effectively.
Let’s begin with the smallest unit of knowledge within the abstraction and work our approach up.
- Event Item: An occasion merchandise is a key-value pair that customers use to retailer information for a given occasion. For instance: {“device_type”: “ios”}.
- Event: An occasion is a structured assortment of a number of such occasion objects. An occasion happens at a particular cut-off date and is recognized by a client-generated timestamp and an occasion identifier (resembling a UUID). This mixture of event_time and event_id additionally kinds a part of the distinctive idempotency key for the occasion, enabling customers to soundly retry requests.
- Time Series ID: A time_series_id is a group of a number of such occasions over the dataset’s retention interval. For occasion, a device_id would retailer all occasions occurring for a given machine over the retention interval. All occasions are immutable, and the TimeSeries service solely ever appends occasions to a given time sequence ID.
- Namespace: A namespace is a group of time sequence IDs and occasion information, representing the whole TimeSeries dataset. Users can create a number of namespaces for every of their use instances. The abstraction applies varied tunable choices on the namespace stage, which we are going to talk about additional once we discover the service’s management airplane.
The abstraction gives the next APIs to work together with the occasion information.
WriteEventRecordsSync: This endpoint writes a batch of occasions and sends again a sturdiness acknowledgement to the consumer. This is utilized in instances the place customers require a assure of sturdiness.
WriteEventRecords: This is the fire-and-forget model of the above endpoint. It enqueues a batch of occasions with out the sturdiness acknowledgement. This is utilized in instances like logging or tracing, the place customers care extra about throughput and may tolerate a small quantity of knowledge loss.
{
"namespace": "my_dataset",
"occasions": [
{
"timeSeriesId": "profile100",
"eventTime": "2024-10-03T21:24:23.988Z",
"eventId": "550e8400-e29b-41d4-a716-446655440000",
"eventItems": [
{
"eventItemKey": "deviceType",
"eventItemValue": "aW9z"
},
{
"eventItemKey": "deviceMetadata",
"eventItemValue": "c29tZSBtZXRhZGF0YQ=="
}
]
},
{
"timeSeriesId": "profile100",
"occasionTime": "2024-10-03T21:23:30.000Z",
"eventId": "123e4567-e89b-12d3-a456-426614174000",
"occasionItems": [
{
"eventItemKey": "deviceType",
"eventItemValue": "YW5kcm9pZA=="
}
]
}
]
}
ReadvertEventRecords: Given a mix of a namespace, a timeSeriesId, a timeInterval, and non-obligatory eventFilters, this endpoint returns all of the matching occasions, sorted descending by event_time, with low millisecond latency.
{
"namespace": "my_dataset",
"timeSeriesId": "profile100",
"timeInterval": {
"begin": "2024-10-02T21:00:00.000Z",
"finish": "2024-10-03T21:00:00.000Z"
},
"eventFilters": [
{
"matchEventItemKey": "deviceType",
"matchEventItemValue": "aW9z"
}
],
"pageSize": 100,
"completeRecordLimit": 1000
}
SearchEventRecords: Given a search standards and a time interval, this endpoint returns all of the matching occasions. These use instances are tremendous with ultimately constant reads.
{
"namespace": "my_dataset",
"timeInterval": {
"begin": "2024-10-02T21:00:00.000Z",
"finish": "2024-10-03T21:00:00.000Z"
},
"searchQuery": {
"booleanQuery": {
"searchQuery": [
{
"equals": {
"eventItemKey": "deviceType",
"eventItemValue": "aW9z"
}
},
{
"range": {
"eventItemKey": "deviceRegistrationTimestamp",
"lowerBound": {
"eventItemValue": "MjAyNC0xMC0wMlQwMDowMDowMC4wMDBa",
"inclusive": true
},
"upperBound": {
"eventItemValue": "MjAyNC0xMC0wM1QwMDowMDowMC4wMDBa"
}
}
}
],
"operator": "AND"
}
},
"pageSize": 100,
"completeRecordLimit": 1000
}
AggregateEventRecords: Given a search standards and an aggregation mode (e.g. DistinctAggregation) , this endpoint performs the given aggregation inside a given time interval. Similar to the Search endpoint, customers can tolerate eventual consistency and a probably larger latency (in seconds).
{
"namespace": "my_dataset",
"timeInterval": {
"begin": "2024-10-02T21:00:00.000Z",
"finish": "2024-10-03T21:00:00.000Z"
},
"searchQuery": {...some search standards...},
"aggregationQuery": {
"distinct": {
"occasionItemKey": "deviceType",
"pageSize": 100
}
}
}
In the next sections, we are going to discuss how we work together with this information on the storage layer.
The storage layer for TimeSeries includes a major information retailer and an non-obligatory index information retailer. The major information retailer ensures information sturdiness throughout writes and is used for major learn operations, whereas the index information retailer is utilized for search and combination operations. At Netflix, Apache Cassandra is the popular selection for storing sturdy information in high-throughput eventualities, whereas Elasticsearch is the popular information retailer for indexing. However, much like our strategy with the API, the storage layer just isn’t tightly coupled to those particular information shops. Instead, we outline storage API contracts that should be fulfilled, permitting us the flexibleness to switch the underlying information shops as wanted.
In this part, we are going to discuss how we leverage Apache Cassandra for TimeSeries use instances.
Partitioning Scheme
At Netflix’s scale, the continual inflow of occasion information can rapidly overwhelm conventional databases. Temporal partitioning addresses this problem by dividing the information into manageable chunks primarily based on time intervals, resembling hourly, every day, or month-to-month home windows. This strategy permits environment friendly querying of particular time ranges with out the necessity to scan the whole dataset. It additionally permits Netflix to archive, compress, or delete older information effectively, optimizing each storage and question efficiency. Additionally, this partitioning mitigates the efficiency points usually related to extensive partitions in Cassandra. By using this technique, we will function at a lot larger disk utilization, because it reduces the necessity to reserve massive quantities of disk house for compactions, thereby saving prices.
Here is what it seems like :
Time Slice: A time slice is the unit of knowledge retention and maps on to a Cassandra desk. We create a number of such time slices, every masking a particular interval of time. An occasion lands in one among these slices primarily based on the event_time. These slices are joined with no time gaps in between, with operations being start-inclusive and end-exclusive, guaranteeing that each one information lands in one of many slices. By using these time slices, we will effectively implement retention by dropping whole tables, which reduces cupboard space and saves on prices.
Why not use row-based Time-To-Live (TTL)?
Using TTL on particular person occasions would generate a major variety of tombstones in Cassandra, degrading efficiency, particularly throughout vary scans. By using discrete time slices and dropping them, we keep away from the tombstone problem solely. The tradeoff is that information could also be retained barely longer than crucial, as a whole desk’s time vary should fall outdoors the retention window earlier than it may be dropped. Additionally, TTLs are tough to regulate later, whereas TimeSeries can lengthen the dataset retention immediately with a single management airplane operation.
Time Buckets: Within a time slice, information is additional partitioned into time buckets. This facilitates efficient vary scans by permitting us to focus on particular time buckets for a given question vary. The tradeoff is that if a person needs to learn the whole vary of knowledge over a big time interval, we should scan many partitions. We mitigate potential latency by scanning these partitions in parallel and aggregating the information on the finish. In most instances, the benefit of concentrating on smaller information subsets outweighs the learn amplification from these scatter-gather operations. Typically, customers learn a smaller subset of knowledge somewhat than the whole retention vary.
Event Buckets: To handle extraordinarily high-throughput write operations, which can end in a burst of writes for a given time sequence inside a brief interval, we additional divide the time bucket into occasion buckets. This prevents overloading the identical partition for a given time vary and in addition reduces partition sizes additional, albeit with a slight enhance in learn amplification.
Note: With Cassandra 4.x onwards, we discover a considerable enchancment within the efficiency of scanning a variety of knowledge in a large partition. See Future Enhancements on the finish to see the Dynamic Event bucketing work that goals to make the most of this.
Storage Tables
We use two sorts of tables
- Data tables: These are the time slices that retailer the precise occasion information.
- Metadata desk: This desk shops details about how every time slice is configured per namespace.
Data tables
The partition key permits splitting occasions for a time_series_id over a variety of time_bucket(s) and event_bucket(s), thus mitigating scorching partitions, whereas the clustering key permits us to maintain information sorted on disk within the order we nearly all the time wish to learn it. The value_metadata column shops metadata for the event_item_value resembling compression.
Writing to the information desk:
User writes will land in a given time slice, time bucket, and occasion bucket as an element of the event_time hooked up to the occasion. This issue is dictated by the management airplane configuration of a given namespace.
For instance:
During this course of, the author makes selections on tips on how to deal with the information earlier than writing, resembling whether or not to compress it. The value_metadata column information any such post-processing actions, guaranteeing that the reader can precisely interpret the information.
Reading from the information desk:
The under illustration depicts at a high-level on how we scatter-gather the reads from a number of partitions and be part of the end result set on the finish to return the ultimate end result.
Metadata desk
This desk shops the configuration information in regards to the time slices for a given namespace.
Note the next:
- No Time Gaps: The end_time of a given time slice overlaps with the start_time of the following time slice, guaranteeing all occasions discover a dwelling.
- Retention: The standing signifies which tables fall inside and outdoors of the retention window.
- Flexible: This metadata will be adjusted per time slice, permitting us to tune the partition settings of future time slices primarily based on noticed information patterns within the present time slice.
There is much more data that may be saved into the metadata column (e.g., compaction settings for the desk), however we solely present the partition settings right here for brevity.
To assist secondary entry patterns through non-primary key attributes, we index information into Elasticsearch. Users can configure a listing of attributes per namespace that they want to search and/or combination information on. The service extracts these fields from occasions as they stream in, indexing the resultant paperwork into Elasticsearch. Depending on the throughput, we might use Elasticsearch as a reverse index, retrieving the complete information from Cassandra, or we might retailer the whole supply information instantly in Elasticsearch.
Note: Again, customers are by no means instantly uncovered to Elasticsearch, identical to they don’t seem to be instantly uncovered to Cassandra. Instead, they work together with the Search and Aggregate API endpoints that translate a given question to that wanted for the underlying datastore.
In the following part, we are going to discuss how we configure these information shops for various datasets.
The information airplane is answerable for executing the learn and write operations, whereas the management airplane configures each side of a namespace’s habits. The information airplane communicates with the TimeSeries management stack, which manages this configuration data. In flip, the TimeSeries management stack interacts with a sharded Data Gateway Platform Control Plane that oversees management configurations for all abstractions and namespaces.
Separating the tasks of the information airplane and management airplane helps preserve the excessive availability of our information airplane, because the management airplane takes on duties that will require some type of schema consensus from the underlying information shops.
The under configuration snippet demonstrates the immense flexibility of the service and the way we will tune a number of issues per namespace utilizing our management airplane.
"persistence_configuration": [
{
"id": "PRIMARY_STORAGE",
"physical_storage": {
"type": "CASSANDRA", // type of primary storage
"cluster": "cass_dgw_ts_tracing", // physical cluster name
"dataset": "tracing_default" // maps to the keyspace
},
"config": {
"timePartition": {
"secondsPerTimeSlice": "129600", // width of a time slice
"secondPerTimeBucket": "3600", // width of a time bucket
"eventBuckets": 4 // how many event buckets within
},
"queueBuffering": {
"coalesce": "1s", // how long to coalesce writes
"bufferCapacity": 4194304 // queue capacity in bytes
},
"consistencyScope": "LOCAL", // single-region/multi-region
"consistencyTarget": "EVENTUAL", // read/write consistency
"acceptLimit": "129600s" // how far back writes are allowed
},
"lifecycleConfigs": {
"lifecycleConfig": [ // Primary store data retention
{
"type": "retention",
"config": {
"close_after": "1296000s", // close for reads/writes
"delete_after": "1382400s" // drop time slice
}
}
]
}
},
{
"id": "INDEX_STORAGE",
"bodilyStorage": {
"sort": "ELASTICSEARCH", // sort of index storage
"cluster": "es_dgw_ts_tracing", // ES cluster title
"dataset": "tracing_default_useast1" // base index title
},
"config": {
"timePartition": {
"secondsPerSlice": "129600" // width of the index slice
},
"consistencyScope": "LOCAL",
"consistencyTarget": "EVENTUAL", // how ought to we learn/write information
"acceptLimit": "129600s", // how far again writes are allowed
"indexConfig": {
"fieldMapping": { // fields to extract to index
"tags.nf.app": "KEYWORD",
"tags.length": "INTEGER",
"tags.enabled": "BOOLEAN"
},
"refreshInterval": "60s" // Index associated settings
}
},
"lifecycleConfigs": {
"lifecycleConfig": [
{
"type": "retention", // Index retention settings
"config": {
"close_after": "1296000s",
"delete_after": "1382400s"
}
}
]
}
}
]
With so many various parameters, we want automated provisioning workflows to infer the most effective settings for a given workload. When customers wish to create their namespaces, they specify a listing of workload wishes, which the automation interprets into concrete infrastructure and associated management airplane configuration. We extremely encourage you to look at this ApacheCon discuss, by one among our beautiful colleagues Joey Lynch, on how we obtain this. We might go into element on this topic in one among our future weblog posts.
Once the system provisions the preliminary infrastructure, it then scales in response to the person workload. The subsequent part describes how that is achieved.
Our customers might function with restricted data on the time of provisioning their namespaces, leading to best-effort provisioning estimates. Further, evolving use-cases might introduce new throughput necessities over time. Here’s how we handle this:
- Horizontal scaling: TimeSeries server cases can auto-scale up and down as per hooked up scaling insurance policies to satisfy the site visitors demand. The storage server capability will be recomputed to accommodate altering necessities utilizing our capability planner.
- Vertical scaling: We might also select to vertically scale our TimeSeries server cases or our storage cases to get higher CPU, RAM and/or hooked up storage capability.
- Scaling disk: We might connect EBS to retailer information if the capability planner prefers infrastructure that provides bigger storage at a decrease value somewhat than SSDs optimized for latency. In such instances, we deploy jobs to scale the EBS quantity when the disk storage reaches a sure proportion threshold.
- Re-partitioning information: Inaccurate workload estimates can result in over or under-partitioning of our datasets. TimeSeries control-plane can regulate the partitioning configuration for upcoming time slices, as soon as we understand the character of knowledge within the wild (through partition histograms). In the longer term we plan to assist re-partitioning of older information and dynamic partitioning of present information.
So far, we have now seen how TimeSeries shops, configures and interacts with occasion datasets. Let’s see how we apply totally different methods to enhance the efficiency of our operations and supply higher ensures.
Event Idempotency
We desire to bake in idempotency in all mutation endpoints, in order that customers can retry or hedge their requests safely. Hedging is when the consumer sends an an identical competing request to the server, if the unique request doesn’t come again with a response in an anticipated period of time. The consumer then responds with whichever request completes first. This is finished to maintain the tail latencies for an utility comparatively low. This can solely be completed safely if the mutations are idempotent. For TimeSeries, the mixture of event_time, event_id and event_item_key kind the idempotency key for a given time_series_id occasion.
SLO-based Hedging
We assign Service Level Objectives (SLO) targets for various endpoints inside TimeSeries, as a sign of what we predict the efficiency of these endpoints ought to be for a given namespace. We can then hedge a request if the response doesn’t come again in that configured period of time.
"slos": {
"learn": { // SLOs per endpoint
"latency": {
"goal": "0.5s", // hedge round this quantity
"max": "1s" // time-out round this quantity
}
},
"write": {
"latency": {
"goal": "0.01s",
"max": "0.05s"
}
}
}
Partial Return
Sometimes, a consumer could also be delicate to latency and prepared to just accept a partial end result set. An actual-world instance of that is real-time frequency capping. Precision just isn’t essential on this case, but when the response is delayed, it turns into virtually ineffective to the upstream consumer. Therefore, the consumer prefers to work with no matter information has been collected thus far somewhat than timing out whereas ready for all the information. The TimeSeries consumer helps partial returns round SLOs for this function. Importantly, we nonetheless preserve the newest order of occasions on this partial fetch.
Adaptive Pagination
All reads begin with a default fanout issue, scanning 8 partition buckets in parallel. However, if the service layer determines that the time_series dataset is dense — i.e., most reads are glad by studying the primary few partition buckets — then it dynamically adjusts the fanout issue of future reads with the intention to cut back the learn amplification on the underlying datastore. Conversely, if the dataset is sparse, we might wish to enhance this restrict with an affordable higher certain.
Limited Write Window
In most instances, the energetic vary for writing information is smaller than the vary for studying information — i.e., we wish a variety of time to grow to be immutable as quickly as attainable in order that we will apply optimizations on high of it. We management this by having a configurable “acceptLimit” parameter that stops customers from writing occasions older than this time restrict. For instance, an settle for restrict of 4 hours implies that customers can not write occasions older than now() — 4 hours. We typically increase this restrict for backfilling historic information, however it’s tuned again down for normal write operations. Once a variety of knowledge turns into immutable, we will safely do issues like caching, compressing, and compacting it for reads.
Buffering Writes
We often leverage this service for dealing with bursty workloads. Rather than overwhelming the underlying datastore with this load , we purpose to distribute it extra evenly by permitting occasions to coalesce over quick durations (usually seconds). These occasions accumulate in in-memory queues operating on every occasion. Dedicated shoppers then steadily drain these queues, grouping the occasions by their partition key, and batching the writes to the underlying datastore.
The queues are tailor-made to every datastore since their operational traits rely on the precise datastore being written to. For occasion, the batch measurement for writing to Cassandra is considerably smaller than that for indexing into Elasticsearch, resulting in totally different drain charges and batch sizes for the related shoppers.
While utilizing in-memory queues does enhance JVM rubbish assortment, we have now skilled substantial enhancements by transitioning to JDK 21 with ZGC. To illustrate the impression, ZGC has lowered our tail latencies by a formidable 86%:
Because we use in-memory queues, we’re liable to dropping occasions in case of an occasion crash. As such, these queues are solely used to be used instances that may tolerate some quantity of knowledge loss .e.g. tracing/logging. For use instances that want assured sturdiness and/or read-after-write consistency, these queues are successfully disabled and writes are flushed to the information retailer nearly instantly.
Dynamic Compaction
Once a time slice exits the energetic write window, we will leverage the immutability of the information to optimize it for learn efficiency. This course of might contain re-compacting immutable information utilizing optimum compaction methods, dynamically shrinking and/or splitting shards to optimize system sources, and different comparable methods to make sure quick and dependable efficiency.
The following part gives a glimpse into the real-world efficiency of a few of our TimeSeries datasets.
The service can write information within the order of low single digit milliseconds
whereas persistently sustaining secure point-read latencies:
At the time of penning this weblog, the service was processing near 15 million occasions/second throughout all of the totally different datasets at peak globally.
The TimeSeries Abstraction performs a significant function throughout key companies at Netflix. Here are some impactful use instances:
- Tracing and Insights: Logs traces throughout all apps and micro-services inside Netflix, to grasp service-to-service communication, assist in debugging of points, and reply assist requests.
- User Interaction Tracking: Tracks hundreds of thousands of person interactions — resembling video playbacks, searches, and content material engagement — offering insights that improve Netflix’s advice algorithms in real-time and enhance the general person expertise.
- Feature Rollout and Performance Analysis: Tracks the rollout and efficiency of recent product options, enabling Netflix engineers to measure how customers interact with options, which powers data-driven selections about future enhancements.
- Asset Impression Tracking and Optimization: Tracks asset impressions guaranteeing content material and belongings are delivered effectively whereas offering real-time suggestions for optimizations.
- Billing and Subscription Management: Stores historic information associated to billing and subscription administration, guaranteeing accuracy in transaction information and supporting customer support inquiries.
and extra…
As the use instances evolve, and the necessity to make the abstraction even more economical grows, we purpose to make many enhancements to the service within the upcoming months. Some of them are:
- Tiered Storage for Cost Efficiency: Support shifting older, lesser-accessed information into cheaper object storage that has larger time to first byte, probably saving Netflix hundreds of thousands of {dollars}.
- Dynamic Event Bucketing: Support real-time partitioning of keys into optimally-sized partitions as occasions stream in, somewhat than having a considerably static configuration on the time of provisioning a namespace. This technique has an enormous benefit of not partitioning time_series_ids that don’t want it, thus saving the general value of learn amplification. Also, with Cassandra 4.x, we have now famous main enhancements in studying a subset of knowledge in a large partition that would lead us to be much less aggressive with partitioning the whole dataset forward of time.
- Caching: Take benefit of immutability of knowledge and cache it intelligently for discrete time ranges.
- Count and different Aggregations: Some customers are solely involved in counting occasions in a given time interval somewhat than fetching all of the occasion information for it.
The TimeSeries Abstraction is an important element of Netflix’s on-line information infrastructure, taking part in a vital function in supporting each real-time and long-term decision-making. Whether it’s monitoring system efficiency throughout high-traffic occasions or optimizing person engagement by means of habits analytics, TimeSeries Abstraction ensures that Netflix operates seamlessly and effectively on a world scale.
As Netflix continues to innovate and develop into new verticals, the TimeSeries Abstraction will stay a cornerstone of our platform, serving to us push the boundaries of what’s attainable in streaming and past.
Stay tuned for Part 2, the place we’ll introduce our Distributed Counter Abstraction, a key component of Netflix’s Composite Abstractions, constructed on high of the TimeSeries Abstraction.
Special because of our beautiful colleagues who contributed to TimeSeries Abstraction’s success: Tom DeVoe Mengqing Wang, Kartik Sathyanarayanan, Jordan West, Matt Lehman, Cheng Wang, Chris Lohfink .