Feature Request: Allow specification of Timestamp Precision #9
Comments
|
Great idea💡 |
|
@packet-rat I would like to get a bit more info about the needs. Is the goal here to have greater precision? Or to have timestamps with 1 and 2 digits for sub-second precision ? such as |
|
I can provide some real world use cases/scenarios:
(1) We are tracking Adversary attacks across a wide set of companies in a "Community of Trust" for a given sector. We are analyzing target lists, the sequencing of attack messages, etc.
Companies in this "Community of Trust" share real time logging data.
Did adversary attack company A, followed by Company B, etc.
How much delay between individual messages, blocks of messages?
Did adversary attack across all companies in a sector concurrently? If so what was the individual targeting pattern?
For Attack Tool foot printing: what was the timing of attack messages?
For Adversary identification what are the temporal patterns of there attacks across a given sector, multiple sectors, etc.
(2) We are tracking Adversary command and control traffic across multiple high speed Internet Egress points to multiple destinations.
Currently scenarios both require microsecond granularity/precision.
Full realization pre-supposses some of the other wish list items for generating specific "real world" events. For now, temporal consistency and timestamp precision for generated data sets allows testing of causality/correlation algorithms.
|
|
Okay I understand the use case. But what is the precision implementation you need to support? You want to generate sample data with timestamps that are close together based on granularity? You want subsecond precision beyond 3 digits? Can you give me a few timestamp examples of the differences in precision you want to generate? |
|
@packet-rat support for this in inbound. The STIX-Java library has a pending update for release that will provide nano-second precision date support. This be released soon in next few days |
|
@packet-rat would you expect to want/need to set precisions on a per field basis? Or more of a global precision? Example if a object has three date fields, would you just want to set "9 digits of precision", or would you want to set precisions for each field? |
|
In practice only certain types of fields require sub-millisecond precision. So one would not generally want to clutter up all timestamps. I think some of this will ultimately be driven by the current "discourse" in the CTI TC community.
I'm going through your code as time permits to figure out* "where the knobs are". If I need to generate timestamps with sub-nansecond precision is this the only module that needs changing:
src/main/java/io/digitalstate/stix/helpers/StixDataFormats.java <StephenOTT/STIX-Java@cb5f6c1#diff-e9bdf0609c89eb5ba1cdd1fc95886951>
* Note: I am a neophyte with Java.
Pat Maroney
On Apr 26, 2019, at 6:01 PM, Stephen Russett <[email protected]> wrote:
@packet-rat <https://github.com/packet-rat> would you expect to want/need to set precisions on a per field basis? Or more of a global precision?
Example if a object has three date fields, would you just want to set "9 digits of precision", or would you want to set precisions for each field?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#9 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAYSFYI364DRJGFCFJNMU4DPSN3SBANCNFSM4HGAOFPA>.
|
|
So my iteration 1 plan would be: Make observables configurable with a lower date (what is the oldest date), and make each date field of each observable configurable for precision (number of digits). So in the rest api, you will provide a probably query params where you can configure each fieldname with digit counts. Likely will also convert this to supporting a Json body so eventually all fields can be configured in different ways. The patch that was merged yesterday into STIX-Java now supports fully configurable date precision. So just need to update Faker with this support to provide configs into the date generators. |
|
@packet-rat check out the first iteration of support for configurable data generation https://github.com/StephenOTT/STIX-Faker/tree/0.8.0#observed-data use the 0.8.0 release. See the https://github.com/StephenOTT/STIX-Faker/blob/master/src/main/groovy/io/digitalstate/stix/faker/configs/ObservedDataGeneratorConfig.groovy for other configuration fields |
|
Hard to keep up with you Stephen! Thanks for everything.:
Can we add a way to specify earliest_time and latest_time to bind the range of the temporally consistent timestamps?
Pat Maroney
|
|
@packet-rat you will have to adjust the various date fields to support the times you want. Each date field is a Instant. So it will support a ISO 8601 String such as See the https://github.com/StephenOTT/STIX-Faker/blob/master/src/main/groovy/io/digitalstate/stix/faker/configs/ObservedDataGeneratorConfig.groovy for the various instant fields. as a example: {
"propCreatedLowerDate": "2019-01-01T12:00:00.000Z",
"propCreatedUpperDate": "2019-01-02T12:00:00.000Z",
"propCreatedProbability": 100,
"propFirstObservedSubsecondPrecision": 9,
"propLastObservedSubsecondPrecision": 6,
"macAddressCoo": {
"occurrence_probability": 100,
"occurs_count_lower": 99,
"occurs_count_upper": 100
}
}Like what you want to be editing is a combination of:
Note: The Lower Date on Last Observed is automatically set for you (the last observed LOWER is the First Observed date. |
|
Thanks for showing me the Knobs!
Patrick Maroney
Chief Architect
CyberIntel-LD Initiative
Email: [email protected]
Cell: (609)841-5104
On Apr 30, 2019, at 5:27 PM, Stephen Russett <[email protected]> wrote:
@packet-rat you will have to adjust the various date fields to support the times you want.
Each date field is a Instant. So it will support a ISO 8601 String such as 2019-01-01T12:00:00.000Z
See the https://github.com/StephenOTT/STIX-Faker/blob/master/src/main/groovy/io/digitalstate/stix/faker/configs/ObservedDataGeneratorConfig.groovy for the various instant fields.
as a example:
{
"propCreatedLowerDate": "2019-01-01T12:00:00.000Z",
"propCreatedUpperDate": "2019-01-02T12:00:00.000Z",
"propCreatedProbability": 100,
"propFirstObservedSubsecondPrecision": 9,
"propLastObservedSubsecondPrecision": 6,
"macAddressCoo": {
"occurrence_probability": 100,
"occurs_count_lower": 99,
"occurs_count_upper": 100
}
}
Like what you want to be editing is a combination of:
propCreatedLowerDate
propCreatedUpperDate
propFirstObservedLowerDate
propFirstObservedUpperDate
propFirstObservedSubsecondPrecision
propLastObservedUpperDate
propLastObservedSubsecondPrecision
Note: The Lower Date on Last Observed is automatically set for you (the last observed LOWER is the First Observed date.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Let me know how it works out |
|
@packet-rat, was just thinking about your use case you raised above: take a look at: https://github.com/StephenOTT/TAXII-springboot-bpmn. The ability to route decisions based on stix data would likely fit nicely with your analysis capabilities. So the analysis is done by your existing worker, and the decision flow and which analysis you want to conduct when / at which step, might be a good fit for you |
|
@packet-rat have you had a chance to test? |
|
Sorry...changed jobs and I’ve been busy.
On May 30, 2019, at 8:18 PM, Stephen Russett <[email protected]> wrote:
@packet-rat have you had a chance to test?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
Allow specification of Timestamp Precision for generated
sightings,observered_data, etc. where assertion on the actual time of the event are made.The text was updated successfully, but these errors were encountered: