hdrhistogram-go – github.com/HdrHistogram/hdrhistogram-go Index | Examples | Files

package hdrhistogram

import "github.com/HdrHistogram/hdrhistogram-go"

Package hdrhistogram provides an implementation of Gil Tene's HDR Histogram data structure. The HDR Histogram allows for fast and accurate analysis of the extreme ranges of data with non-normal distributions, like latency.

Histograms are encoded using the HdrHistogram V2 format which is based on an adapted ZigZag LEB128 encoding where: consecutive zero counters are encoded as a negative number representing the count of consecutive zeros non zero counter values are encoded as a positive number A typical histogram (2 digits precision 1 usec to 1 day range) can be encoded in less than the typical MTU size of 1500 bytes.

The log format encodes into a single file, multiple histograms with optional shared meta data.

Index

Examples

Constants 

const (
	V2EncodingCookieBase           int32 = 0x1c849303
	V2CompressedEncodingCookieBase int32 = 0x1c849304

	ENCODING_HEADER_SIZE = 40
)
const HISTOGRAM_LOG_FORMAT_VERSION = "1.3"
const MsToNsRatio float64 = 1000000.0

Types

type Bar 

type Bar struct {
	From, To, Count int64
}

Histogram bar for plotting

func (Bar) String 

func (b Bar) String() string

Pretty print as csv for easy plotting

type Bracket 

type Bracket struct {
	Quantile       float64
	Count, ValueAt int64
}

A Bracket is a part of a cumulative distribution.

type Histogram 

type Histogram struct {
	// contains filtered or unexported fields
}

A Histogram is a lossy data structure used to record the distribution of non-normally distributed data (like latency) with a high degree of accuracy and a bounded degree of precision.

func Decode 

func Decode(encoded []byte) (rh *Histogram, err error)

Decode returns a new Histogram by decoding it from a String containing a base64 encoded compressed histogram representation.

func Import 

func Import(s *Snapshot) *Histogram

Import returns a new Histogram populated from the Snapshot data (which the caller must stop accessing).

func New 

func New(lowestDiscernibleValue, highestTrackableValue int64, numberOfSignificantValueDigits int) *Histogram

Construct a Histogram given the Lowest and Highest values to be tracked and a number of significant decimal digits.

Providing a lowestDiscernibleValue is useful in situations where the units used for the histogram's values are much smaller that the minimal accuracy required. E.g. when tracking time values stated in nanosecond units, where the minimal accuracy required is a microsecond, the proper value for lowestDiscernibleValue would be 1000.

Note: the numberOfSignificantValueDigits must be [1,5]. If lower than 1 the numberOfSignificantValueDigits will be forced to 1, and if higher than 5 the numberOfSignificantValueDigits will be forced to 5.

Example

This latency Histogram could be used to track and analyze the counts of observed integer values between 1 us and 30000000 us ( 30 secs ) while maintaining a value precision of 4 significant digits across that range, translating to a value resolution of :

  • 1 microsecond up to 10 milliseconds,
  • 100 microsecond (or better) from 10 milliseconds up to 10 seconds,
  • 300 microsecond (or better) from 10 seconds up to 30 seconds,

nolint

Code: 

{
	lH := hdrhistogram.New(1, 30000000, 4)
	input := []int64{
		459876, 669187, 711612, 816326, 931423, 1033197, 1131895, 2477317,
		3964974, 12718782,
	}

	for _, sample := range input {
		lH.RecordValue(sample)
	}

	fmt.Printf("Percentile 50: %d\n", lH.ValueAtQuantile(50.0))

	// Output:
	// Percentile 50: 931423
}

Output:

Percentile 50: 931423

func (*Histogram) ByteSize 

func (h *Histogram) ByteSize() int

ByteSize returns an estimate of the amount of memory allocated to the histogram in bytes.

N.B.: This does not take into account the overhead for slices, which are small, constant, and specific to the compiler version.

func (*Histogram) CumulativeDistribution 

func (h *Histogram) CumulativeDistribution() []Bracket

CumulativeDistribution returns an ordered list of brackets of the distribution of recorded values.

func (*Histogram) CumulativeDistributionWithTicks 

func (h *Histogram) CumulativeDistributionWithTicks(ticksPerHalfDistance int32) []Bracket

CumulativeDistribution returns an ordered list of brackets of the distribution of recorded values.

func (*Histogram) Distribution 

func (h *Histogram) Distribution() (result []Bar)

Distribution returns an ordered list of bars of the distribution of recorded values, counts can be normalized to a probability

func (*Histogram) Encode 

func (h *Histogram) Encode(version int32) (buffer []byte, err error)

Encode returns a snapshot view of the Histogram. The snapshot is compact binary representations of the state of the histogram. They are intended to be used for archival or transmission to other systems for further analysis.

func (*Histogram) EndTimeMs 

func (h *Histogram) EndTimeMs() int64

func (*Histogram) Equals 

func (h *Histogram) Equals(other *Histogram) bool

Equals returns true if the two Histograms are equivalent, false if not.

func (*Histogram) Export 

func (h *Histogram) Export() *Snapshot

Export returns a snapshot view of the Histogram. This can be later passed to Import to construct a new Histogram with the same state.

func (*Histogram) HighestTrackableValue 

func (h *Histogram) HighestTrackableValue() int64

HighestTrackableValue returns the upper bound on values that will be added to the histogram

func (*Histogram) LowestTrackableValue 

func (h *Histogram) LowestTrackableValue() int64

LowestTrackableValue returns the lower bound on values that will be added to the histogram

func (*Histogram) Max 

func (h *Histogram) Max() int64

Max returns the approximate maximum recorded value.

func (*Histogram) Mean 

func (h *Histogram) Mean() float64

Mean returns the approximate arithmetic mean of the recorded values.

func (*Histogram) Merge 

func (h *Histogram) Merge(from *Histogram) (dropped int64)

Merge merges the data stored in the given histogram with the receiver, returning the number of recorded values which had to be dropped.

func (*Histogram) Min 

func (h *Histogram) Min() int64

Min returns the approximate minimum recorded value.

func (*Histogram) PercentilesPrint 

func (h *Histogram) PercentilesPrint(writer io.Writer, ticksPerHalfDistance int32, valueScale float64) (outputWriter io.Writer, err error)

Output the percentiles distribution in a text format

Example

The following example details the creation of an histogram used to track and analyze the counts of observed integer values between 0 us and 30000000 us ( 30 secs ) and the printing of the percentile output format nolint

Code: 

{
	lH := hdrhistogram.New(1, 30000000, 3)
	input := []int64{
		459876, 669187, 711612, 816326, 931423, 1033197, 1131895, 2477317,
		3964974, 12718782,
	}

	for _, sample := range input {
		lH.RecordValue(sample)
	}

	lH.PercentilesPrint(os.Stdout, 1, 1.0)
	// Output:
	//  Value	Percentile	TotalCount	1/(1-Percentile)
	//
	//   460031.000     0.000000            1         1.00
	//   931839.000     0.500000            5         2.00
	//  2478079.000     0.750000            8         4.00
	//  3966975.000     0.875000            9         8.00
	// 12722175.000     0.937500           10        16.00
	// 12722175.000     1.000000           10          inf
	// #[Mean    =  2491481.600, StdDeviation   =  3557920.109]
	// #[Max     = 12722175.000, Total count    =           10]
	// #[Buckets =           15, SubBuckets     =         2048]
}

Output:

 Value	Percentile	TotalCount	1/(1-Percentile)

  460031.000     0.000000            1         1.00
  931839.000     0.500000            5         2.00
 2478079.000     0.750000            8         4.00
 3966975.000     0.875000            9         8.00
12722175.000     0.937500           10        16.00
12722175.000     1.000000           10          inf
#[Mean    =  2491481.600, StdDeviation   =  3557920.109]
#[Max     = 12722175.000, Total count    =           10]
#[Buckets =           15, SubBuckets     =         2048]

func (*Histogram) RecordCorrectedValue 

func (h *Histogram) RecordCorrectedValue(v, expectedInterval int64) error

RecordCorrectedValue records the given value, correcting for stalls in the recording process. This only works for processes which are recording values at an expected interval (e.g., doing jitter analysis). Processes which are recording ad-hoc values (e.g., latency for incoming requests) can't take advantage of this.

func (*Histogram) RecordValue 

func (h *Histogram) RecordValue(v int64) error

RecordValue records the given value, returning an error if the value is out of range.

Example

This latency Histogram could be used to track and analyze the counts of observed integer values between 0 us and 30000000 us ( 30 secs ) while maintaining a value precision of 3 significant digits across that range, translating to a value resolution of :

  • 1 microsecond up to 1 millisecond,
  • 1 millisecond (or better) up to one second,
  • 1 second (or better) up to it's maximum tracked value ( 30 seconds ).

nolint

Code: 

{
	lH := hdrhistogram.New(1, 30000000, 3)
	input := []int64{
		459876, 669187, 711612, 816326, 931423, 1033197, 1131895, 2477317,
		3964974, 12718782,
	}

	for _, sample := range input {
		lH.RecordValue(sample)
	}

	fmt.Printf("Percentile 50: %d\n", lH.ValueAtQuantile(50.0))

	// Output:
	// Percentile 50: 931839
}

Output:

Percentile 50: 931839

func (*Histogram) RecordValues 

func (h *Histogram) RecordValues(v, n int64) error

RecordValues records n occurrences of the given value, returning an error if the value is out of range.

func (*Histogram) Reset 

func (h *Histogram) Reset()

Reset deletes all recorded values and restores the histogram to its original state.

func (*Histogram) SetEndTimeMs 

func (h *Histogram) SetEndTimeMs(endTimeMs int64)

func (*Histogram) SetStartTimeMs 

func (h *Histogram) SetStartTimeMs(startTimeMs int64)

func (*Histogram) SetTag 

func (h *Histogram) SetTag(tag string)

func (*Histogram) SignificantFigures 

func (h *Histogram) SignificantFigures() int64

SignificantFigures returns the significant figures used to create the histogram

func (*Histogram) StartTimeMs 

func (h *Histogram) StartTimeMs() int64

func (*Histogram) StdDev 

func (h *Histogram) StdDev() float64

StdDev returns the approximate standard deviation of the recorded values.

func (*Histogram) Tag 

func (h *Histogram) Tag() string

func (*Histogram) TotalCount 

func (h *Histogram) TotalCount() int64

TotalCount returns total number of values recorded.

func (*Histogram) ValueAtPercentile 

func (h *Histogram) ValueAtPercentile(percentile float64) int64

ValueAtPercentile returns the largest value that (100% - percentile) of the overall recorded value entries in the histogram are either larger than or equivalent to.

The passed percentile must be a float64 value in [0.0 .. 100.0] Note that two values are "equivalent" if `ValuesAreEquivalent(value1,value2)` would return true.

Returns 0 if no recorded values exist.

func (*Histogram) ValueAtPercentiles 

func (h *Histogram) ValueAtPercentiles(percentiles []float64) (values map[float64]int64)

ValueAtPercentiles, given an slice of percentiles returns a map containing for each passed percentile, the largest value that (100% - percentile) of the overall recorded value entries in the histogram are either larger than or equivalent to.

Each element in the given an slice of percentiles must be a float64 value in [0.0 .. 100.0] Note that two values are "equivalent" if `ValuesAreEquivalent(value1,value2)` would return true.

Returns a map of 0's if no recorded values exist.

Example

When doing an percentile analysis we normally require more than one percentile to be calculated for the given histogram.

When that is the case ValueAtPercentiles() will deeply optimize the total time to retrieve the percentiles vs the other option which is multiple calls to ValueAtQuantile().

nolint

Code: 

{
	histogram := hdrhistogram.New(1, 30000000, 3)

	for i := 0; i < 1000000; i++ {
		histogram.RecordValue(int64(i))
	}

	percentileValuesMap := histogram.ValueAtPercentiles([]float64{50.0, 95.0, 99.0, 99.9})
	fmt.Printf("Percentile 50: %d\n", percentileValuesMap[50.0])
	fmt.Printf("Percentile 95: %d\n", percentileValuesMap[95.0])
	fmt.Printf("Percentile 99: %d\n", percentileValuesMap[99.0])
	fmt.Printf("Percentile 99.9: %d\n", percentileValuesMap[99.9])

	// Output:
	// Percentile 50: 500223
	// Percentile 95: 950271
	// Percentile 99: 990207
	// Percentile 99.9: 999423

}

Output:

Percentile 50: 500223
Percentile 95: 950271
Percentile 99: 990207
Percentile 99.9: 999423

func (*Histogram) ValueAtQuantile 

func (h *Histogram) ValueAtQuantile(q float64) int64

ValueAtQuantile returns the largest value that (100% - percentile) of the overall recorded value entries in the histogram are either larger than or equivalent to.

The passed quantile must be a float64 value in [0.0 .. 100.0] Note that two values are "equivalent" if `ValuesAreEquivalent(value1,value2)` would return true.

Returns 0 if no recorded values exist.

func (*Histogram) ValuesAreEquivalent 

func (h *Histogram) ValuesAreEquivalent(value1, value2 int64) (result bool)

Determine if two values are equivalent with the histogram's resolution. Where "equivalent" means that value samples recorded for any two equivalent values are counted in a common total count.

type HistogramLogOptions 

type HistogramLogOptions struct {
	// contains filtered or unexported fields
}

func DefaultHistogramLogOptions 

func DefaultHistogramLogOptions() *HistogramLogOptions

type HistogramLogReader 

type HistogramLogReader struct {
	// contains filtered or unexported fields
}

func NewHistogramLogReader 

func NewHistogramLogReader(log io.Reader) *HistogramLogReader
Example

The log format encodes into a single file, multiple histograms with optional shared meta data. The following example showcases reading a log file into a slice of histograms nolint

Code:play  

package main

import (
	"bytes"
	"fmt"

	hdrhistogram "github.com/HdrHistogram/hdrhistogram-go"
	"io/ioutil"
)

func main() {
	dat, _ := ioutil.ReadFile("./test/tagged-Log.logV2.hlog")
	r := bytes.NewReader(dat)

	// Create a histogram log reader
	reader := hdrhistogram.NewHistogramLogReader(r)
	var histograms []*hdrhistogram.Histogram = make([]*hdrhistogram.Histogram, 0)

	// Read all histograms in the file
	for hist, err := reader.NextIntervalHistogram(); hist != nil && err == nil; hist, err = reader.NextIntervalHistogram() {
		histograms = append(histograms, hist)
	}
	fmt.Printf("Read a total of %d histograms\n", len(histograms))

	min := reader.RangeObservedMin()
	max := reader.RangeObservedMax()
	sigdigits := 3
	overallHistogram := hdrhistogram.New(min, max, sigdigits)

	//// We can then merge all histograms into one and retrieve overall metrics
	for _, hist := range histograms {
		overallHistogram.Merge(hist)
	}
	fmt.Printf("Overall count: %d samples\n", overallHistogram.TotalCount())
	fmt.Printf("Overall Percentile 50: %d\n", overallHistogram.ValueAtQuantile(50.0))

}

Output:

Read a total of 42 histograms
Overall count: 32290 samples
Overall Percentile 50: 344319

func (*HistogramLogReader) NextIntervalHistogram 

func (hlr *HistogramLogReader) NextIntervalHistogram() (histogram *Histogram, err error)

func (*HistogramLogReader) NextIntervalHistogramWithRange 

func (hlr *HistogramLogReader) NextIntervalHistogramWithRange(rangeStartTimeSec, rangeEndTimeSec float64, absolute bool) (histogram *Histogram, err error)

func (*HistogramLogReader) ObservedMax 

func (hlr *HistogramLogReader) ObservedMax() bool

func (*HistogramLogReader) ObservedMin 

func (hlr *HistogramLogReader) ObservedMin() bool

func (*HistogramLogReader) RangeObservedMax 

func (hlr *HistogramLogReader) RangeObservedMax() int64

Returns the overall observed max limit ( up to the current point ) of the read histograms

func (*HistogramLogReader) RangeObservedMin 

func (hlr *HistogramLogReader) RangeObservedMin() int64

Returns the overall observed min limit ( up to the current point ) of the read histograms

type HistogramLogWriter 

type HistogramLogWriter struct {
	// contains filtered or unexported fields
}

func NewHistogramLogWriter 

func NewHistogramLogWriter(log io.Writer) *HistogramLogWriter
Example

The log format encodes into a single file, multiple histograms with optional shared meta data. The following example showcases writing multiple histograms into a log file and then processing them again to confirm a proper encode-decode flow nolint

Code:play  

package main

import (
	"bytes"
	"fmt"

	hdrhistogram "github.com/HdrHistogram/hdrhistogram-go"
	"io/ioutil"
)

func main() {
	var buff bytes.Buffer

	// Create a histogram log writer to write to a bytes.Buffer
	writer := hdrhistogram.NewHistogramLogWriter(&buff)

	writer.OutputLogFormatVersion()
	writer.OutputStartTime(0)
	writer.OutputLegend()

	// Lets create 3 distinct histograms to exemply the logwriter features
	// each one with a time-frame of 60 secs ( 60000 ms )
	hist1 := hdrhistogram.New(1, 30000000, 3)
	hist1.SetStartTimeMs(0)
	hist1.SetEndTimeMs(60000)
	for _, sample := range []int64{10, 20, 30, 40} {
		hist1.RecordValue(sample)
	}
	hist2 := hdrhistogram.New(1, 3000, 3)
	hist1.SetStartTimeMs(60001)
	hist1.SetEndTimeMs(120000)
	for _, sample := range []int64{50, 70, 80, 60} {
		hist2.RecordValue(sample)
	}
	hist3 := hdrhistogram.New(1, 30000, 3)
	hist1.SetStartTimeMs(120001)
	hist1.SetEndTimeMs(180000)
	for _, sample := range []int64{90, 100} {
		hist3.RecordValue(sample)
	}
	writer.OutputIntervalHistogram(hist1)
	writer.OutputIntervalHistogram(hist2)
	writer.OutputIntervalHistogram(hist3)

	ioutil.WriteFile("example.logV2.hlog", buff.Bytes(), 0644)

	// read check
	// Lets read all again and confirm that the total sample count is 10
	dat, _ := ioutil.ReadFile("example.logV2.hlog")
	r := bytes.NewReader(dat)

	// Create a histogram log reader
	reader := hdrhistogram.NewHistogramLogReader(r)
	var histograms []*hdrhistogram.Histogram = make([]*hdrhistogram.Histogram, 0)

	// Read all histograms in the file
	for hist, err := reader.NextIntervalHistogram(); hist != nil && err == nil; hist, err = reader.NextIntervalHistogram() {
		histograms = append(histograms, hist)
	}
	fmt.Printf("Read a total of %d histograms\n", len(histograms))

	min := reader.RangeObservedMin()
	max := reader.RangeObservedMax()
	sigdigits := 3
	overallHistogram := hdrhistogram.New(min, max, sigdigits)

	//// We can then merge all histograms into one and retrieve overall metrics
	for _, hist := range histograms {
		overallHistogram.Merge(hist)
	}
	fmt.Printf("Overall count: %d samples\n", overallHistogram.TotalCount())
}

Output:

Read a total of 3 histograms
Overall count: 10 samples

func (*HistogramLogWriter) BaseTime 

func (lw *HistogramLogWriter) BaseTime() int64

Return the current base time offset

func (*HistogramLogWriter) OutputBaseTime 

func (lw *HistogramLogWriter) OutputBaseTime(base_time_msec int64) (err error)

Log a base time in the log. Base time is represented as seconds since epoch with up to 3 decimal places. Line starts with the leading text '#[BaseTime:'

func (*HistogramLogWriter) OutputComment 

func (lw *HistogramLogWriter) OutputComment(comment string) (err error)

Log a comment to the log. A comment is any line that leads with '#' that is not matched by the BaseTime or StartTime formats. Comments are ignored when parsed.

func (*HistogramLogWriter) OutputIntervalHistogram 

func (lw *HistogramLogWriter) OutputIntervalHistogram(histogram *Histogram) (err error)

Output an interval histogram, using the start/end timestamp indicated in the histogram, and the [optional] tag associated with the histogram. The histogram start and end timestamps are assumed to be in msec units

By convention, histogram start/end time are generally stamped with absolute times in msec since the epoch. For logging with absolute time stamps, the base time would remain zero ( default ). For logging with relative time stamps (time since a start point), the base time should be set with SetBaseTime(baseTime int64)

The max value in the histogram will be reported scaled down by a default maxValueUnitRatio of 1000000.0 (which is the msec : nsec ratio). If you need to specify a different start/end timestamp or a different maxValueUnitRatio you should use OutputIntervalHistogramWithLogOptions(histogram *Histogram, logOptions *HistogramLogOptions)

func (*HistogramLogWriter) OutputIntervalHistogramWithLogOptions 

func (lw *HistogramLogWriter) OutputIntervalHistogramWithLogOptions(histogram *Histogram, logOptions *HistogramLogOptions) (err error)

Output an interval histogram, with the given timestamp information and the [optional] tag associated with the histogram

If you specify non-nil logOptions, and non-zero start timestamp, the the specified timestamp information will be used, and the start timestamp information in the actual histogram will be ignored. If you specify non-nil logOptions, and non-zero start timestamp, the the specified timestamp information will be used, and the end timestamp information in the actual histogram will be ignored. If you specify non-nil logOptions, The max value reported with the interval line will be scaled by the given maxValueUnitRatio, otherwise a default maxValueUnitRatio of 1,000,000 (which is the msec : nsec ratio) will be used.

By convention, histogram start/end time are generally stamped with absolute times in msec since the epoch. For logging with absolute time stamps, the base time would remain zero ( default ). For logging with relative time stamps (time since a start point), the base time should be set with SetBaseTime(baseTime int64)

func (*HistogramLogWriter) OutputLegend 

func (lw *HistogramLogWriter) OutputLegend() (err error)

Output a legend line to the log. Human readable column headers. Ignored when parsed.

func (*HistogramLogWriter) OutputLogFormatVersion 

func (lw *HistogramLogWriter) OutputLogFormatVersion() (err error)

Output a log format version to the log.

func (*HistogramLogWriter) OutputStartTime 

func (lw *HistogramLogWriter) OutputStartTime(start_time_msec int64) (err error)

Log a start time in the log. Start time is represented as seconds since epoch with up to 3 decimal places. Line starts with the leading text '#[StartTime:'

func (*HistogramLogWriter) SetBaseTime 

func (lw *HistogramLogWriter) SetBaseTime(baseTime int64)

Set a base time to subtract from supplied histogram start/end timestamps when logging based on histogram timestamps. baseTime is expected to be in msec since the epoch, as histogram start/end times are typically stamped with absolute times in msec since the epoch.

type Snapshot 

type Snapshot struct {
	LowestTrackableValue  int64
	HighestTrackableValue int64
	SignificantFigures    int64
	Counts                []int64
}

A Snapshot is an exported view of a Histogram, useful for serializing them. A Histogram can be constructed from it by passing it to Import.

type WindowedHistogram 

type WindowedHistogram struct {
	Current *Histogram
	// contains filtered or unexported fields
}

A WindowedHistogram combines histograms to provide windowed statistics.

func NewWindowed 

func NewWindowed(n int, minValue, maxValue int64, sigfigs int) *WindowedHistogram

NewWindowed creates a new WindowedHistogram with N underlying histograms with the given parameters.

func (*WindowedHistogram) Merge 

func (w *WindowedHistogram) Merge() *Histogram

Merge returns a histogram which includes the recorded values from all the sections of the window.

func (*WindowedHistogram) Rotate 

func (w *WindowedHistogram) Rotate()

Rotate resets the oldest histogram and rotates it to be used as the current histogram.

Source Files

hdr.go hdr_encoding.go log_reader.go log_writer.go window.go zigzag.go

Version
v1.1.2 (latest)
Published
Aug 24, 2021
Platform
linux/amd64
Imports
15 packages
Last checked
14 hours ago –

Tools for package owners.