pkg/ directory
Directories ΒΆ
| Path | Synopsis |
|---|---|
| go/pkg/beam | Package beam is an implementation of the Apache Beam (https://beam.apache.org) programming model in Go. |
| go/pkg/beam/artifact | Package artifact contains utilities for staging and retrieving artifacts. |
| go/pkg/beam/artifact/gcsproxy | Package gcsproxy contains artifact staging and retrieval servers backed by GCS. |
| go/pkg/beam/core | Package core contains constants and other static data related to the SDK, such as the SDK Name and version. |
| go/pkg/beam/core/funcx | Package funcx contains functions and types used to perform type analysis of Beam functions. |
| go/pkg/beam/core/graph | Package graph is the internal representation of the Beam execution plan. |
| go/pkg/beam/core/graph/coder | Package coder contains coder representation and utilities. |
| go/pkg/beam/core/graph/coder/testutil | Package testutil contains helpers to test and validate custom Beam Schema coders. |
| go/pkg/beam/core/graph/mtime | Package mtime contains a millisecond representation of time. |
| go/pkg/beam/core/graph/window | Package window contains window representation, windowing strategies and utilities. |
| go/pkg/beam/core/graph/window/trigger | Package trigger helps construct aggregation triggers with beam.WindowInto. |
| go/pkg/beam/core/metrics | Package metrics implements the Beam metrics API, described at http://s.apache.org/beam-metrics-api |
| go/pkg/beam/core/runtime | Package runtime contains runtime hooks and utilities for pipeline options and type registration. |
| go/pkg/beam/core/runtime/coderx | Package coderx contains coders for primitive types that aren't included in the beam model. |
| go/pkg/beam/core/runtime/contextreg | Package contextreg contains the global registrations of functions for extracting ptransform annotations or environment resource hints from context.Context attached to scopes. |
| go/pkg/beam/core/runtime/exec | Package exec contains runtime plan representation and execution. |
| go/pkg/beam/core/runtime/exec/optimized | Package optimized contains type-specialized shims for faster execution. |
| go/pkg/beam/core/runtime/genx | Package genx is a convenience package to better support the code generator. |
| go/pkg/beam/core/runtime/graphx | Package graphx provides facilities to help with the serialization of pipelines into a serializable graph structure suitable for the worker. |
| go/pkg/beam/core/runtime/graphx/schema | Package schema contains utility functions for relating Go types and Beam Schemas. |
| go/pkg/beam/core/runtime/graphx/v1 | |
| go/pkg/beam/core/runtime/harness | Package harness implements the SDK side of the Beam FnAPI. |
| go/pkg/beam/core/runtime/harness/init | Package init contains the harness initialization code defined by the FnAPI. |
| go/pkg/beam/core/runtime/harness/statecache | Package statecache implements the state caching feature described by the Beam Fn API |
| go/pkg/beam/core/runtime/metricsx | |
| go/pkg/beam/core/runtime/pipelinex | Package pipelinex contains utilities for manipulating Beam proto pipelines. |
| go/pkg/beam/core/runtime/xlangx | Package xlangx contains various low-level utilities needed for adding cross-language transforms to the pipeline. |
| go/pkg/beam/core/runtime/xlangx/expansionx | Package expansionx contains utilities for starting expansion services for cross-language transforms. |
| go/pkg/beam/core/sdf | Package contains interfaces used specifically for splittable DoFns. |
| go/pkg/beam/core/state | Package state contains structs for reading and manipulating pipeline state. |
| go/pkg/beam/core/timers | Package timers contains structs for setting pipeline timers. |
| go/pkg/beam/core/typex | Package typex contains full type representation for PCollections and DoFns, and utilities for type checking. |
| go/pkg/beam/core/util | |
| go/pkg/beam/core/util/dot | Package dot produces DOT graphs from Beam graph representations. |
| go/pkg/beam/core/util/hooks | Package hooks allows runners to tailor execution of the worker harness. |
| go/pkg/beam/core/util/ioutilx | Package ioutilx contains additional io utilities. |
| go/pkg/beam/core/util/jsonx | Package jsonx contains utilities for working with JSON encoded data. |
| go/pkg/beam/core/util/protox | Package protox contains utilities for working with protobufs. |
| go/pkg/beam/core/util/reflectx | Package reflectx contains a set of reflection utilities and well-known types. |
| go/pkg/beam/core/util/symtab | Package symtab allows reading low-level symbol information from the symbol table. |
| go/pkg/beam/internal | |
| go/pkg/beam/io | |
| go/pkg/beam/io/avroio | Package avroio contains transforms for reading and writing avro files. |
| go/pkg/beam/io/bigqueryio | Package bigqueryio provides transformations and utilities to interact with Google BigQuery. |
| go/pkg/beam/io/bigtableio | Package bigtableio provides transformations and utilities to interact with Google Bigtable. |
| go/pkg/beam/io/databaseio | Package databaseio provides transformations and utilities to interact with generic database database/sql API. |
| go/pkg/beam/io/datastoreio | Package datastoreio provides transformations and utilities to interact with Google Datastore. |
| go/pkg/beam/io/fhirio | Package fhirio provides an API for reading and writing resources to Google Cloud Healthcare Fhir stores. |
| go/pkg/beam/io/fileio | Package fileio provides transforms for matching and reading files. |
| go/pkg/beam/io/filesystem | Package filesystem contains an extensible file system abstraction. |
| go/pkg/beam/io/filesystem/gcs | Package gcs contains a Google Cloud Storage (GCS) implementation of the Beam file system. |
| go/pkg/beam/io/filesystem/local | Package local contains a local file implementation of the Beam file system. |
| go/pkg/beam/io/filesystem/memfs | Package memfs contains a in-memory Beam filesystem. |
| go/pkg/beam/io/filesystem/s3 | Package s3 contains an AWS S3 implementation of the Beam file system. |
| go/pkg/beam/io/mongodbio | Package mongodbio contains transforms for reading from and writing to MongoDB. |
| go/pkg/beam/io/natsio | Package natsio contains transforms for interacting with NATS. |
| go/pkg/beam/io/parquetio | Package parquetio contains transforms for reading and writing parquet files |
| go/pkg/beam/io/pubsubio | Package pubsubio provides access to Pub/Sub on Dataflow streaming. |
| go/pkg/beam/io/rtrackers | |
| go/pkg/beam/io/rtrackers/offsetrange | Package offsetrange defines a restriction and restriction tracker for offset ranges. |
| go/pkg/beam/io/spannerio | Package spannerio provides an API for reading and writing resouces to Google Spanner datastores. |
| go/pkg/beam/io/synthetic | Package synthetic contains transforms for creating synthetic pipelines. |
| go/pkg/beam/io/textio | Package textio contains transforms for reading and writing text files. |
| go/pkg/beam/io/xlang | |
| go/pkg/beam/io/xlang/bigqueryio | Package bigqueryio contains cross-language functionality for using Google Cloud BigQuery (https://cloud.google.com/bigquery). |
| go/pkg/beam/io/xlang/bigtableio | Package bigtableio contains cross-language functionality for using Google Cloud BigQuery (https://cloud.google.com/bigquery). |
| go/pkg/beam/io/xlang/debeziumio | Package debeziumio contains cross-language functionality for using Debezium (http://kafka.apache.org/). |
| go/pkg/beam/io/xlang/jdbcio | Package jdbcio contains cross-language functionality for reading and writing data to JDBC. |
| go/pkg/beam/io/xlang/kafkaio | Package kafkaio contains cross-language functionality for using Apache Kafka (http://kafka.apache.org/). |
| go/pkg/beam/io/xlang/schemaio | Package schemaio contains utilities for constructing cross-language IO wrappers meant to interface with the Java SDK's Schema IOs. |
| go/pkg/beam/log | Package log contains a re-targetable context-aware logging system. |
| go/pkg/beam/model | Package model contains the portable Beam model contracts. |
| go/pkg/beam/model/fnexecution_v1 | |
| go/pkg/beam/model/jobmanagement_v1 | |
| go/pkg/beam/model/pipeline_v1 | |
| go/pkg/beam/options | |
| go/pkg/beam/options/gcpopts | Package gcpopts contains shared options for Google Cloud Platform. |
| go/pkg/beam/options/jobopts | Package jobopts contains shared options for job submission. |
| go/pkg/beam/options/resource | Package resource supports Beam resource hints to specify scoped hints or annotations to pipelines. |
| go/pkg/beam/provision | Package provision contains utilities for obtaining runtime provision, information -- such as pipeline options. |
| go/pkg/beam/register | Package register contains functions for registering and optimizing your DoFn. |
| go/pkg/beam/runners | Package runners defines the common "--runner" flag. |
| go/pkg/beam/runners/dataflow | Package dataflow contains the Dataflow runner for submitting pipelines to Google Cloud Dataflow. |
| go/pkg/beam/runners/dataflow/dataflowlib | Package dataflowlib translates a Beam pipeline model to the Dataflow API job model, for submission to Google Cloud Dataflow. |
| go/pkg/beam/runners/direct | Package direct contains the direct runner for running single-bundle pipelines in the current process. |
| go/pkg/beam/runners/dot | Package dot is a Beam runner that "runs" a pipeline by producing a DOT graph of the execution plan. |
| go/pkg/beam/runners/flink | Package flink contains the Flink runner. |
| go/pkg/beam/runners/prism | Package prism contains a local runner for running pipelines in the current process. |
| go/pkg/beam/runners/prism/internal | Package internal is where the less separable parts of the runner are put together in order to execute pipelines, and validate that beam features are implemented, and configured appropriately for the variant a pipeline is using. |
| go/pkg/beam/runners/samza | Package samza contains the Samza runner. |
| go/pkg/beam/runners/spark | Package spark contains the Spark runner. |
| go/pkg/beam/runners/universal | Package universal contains a general-purpose runner that can submit jobs to any portable Beam runner. |
| go/pkg/beam/runners/universal/extworker | Package extworker provides an external worker service and related utilities. |
| go/pkg/beam/runners/universal/runnerlib | Package runnerlib contains utilities for submitting Go pipelines to a Beam model runner. |
| go/pkg/beam/runners/vet | Package vet is a Beam runner that "runs" a pipeline by producing generated code to avoid symbol table lookups and reflection in pipeline execution. |
| go/pkg/beam/runners/vet/testpipeline | Package testpipeline exports small test pipelines for testing the vet runner. |
| go/pkg/beam/testing | |
| go/pkg/beam/testing/passert | Package passert contains verification transformations for testing pipelines. |
| go/pkg/beam/testing/ptest | Package ptest contains utilities for pipeline unit testing. |
| go/pkg/beam/testing/teststream | Package teststream contains code configuring the TestStream primitive for use in testing code that is meant to be run on streaming data sources. |
| go/pkg/beam/transforms | |
| go/pkg/beam/transforms/filter | Package filter contains transformations for removing pipeline elements based on various conditions. |
| go/pkg/beam/transforms/periodic | Package periodic contains transformations for generating periodic sequences. |
| go/pkg/beam/transforms/sql | Package sql contains SQL transform APIs, allowing SQL queries to be used in Beam Go pipelines. |
| go/pkg/beam/transforms/sql/sqlx | Package sqlx contains "internal" SQL transform interfaces that are needed by the SQL expansion providers. |
| go/pkg/beam/transforms/stats | Package stats contains transforms for statistical processing. |
| go/pkg/beam/transforms/top | Package top contains transformations for finding the smallest (or largest) N elements based on arbitrary orderings. |
| go/pkg/beam/transforms/xlang | Package xlang contains cross-language transforms. |
| go/pkg/beam/transforms/xlang/dataframe | Package dataframe is a wrapper for DataframeTransform defined in Apache Beam Python SDK. |
| go/pkg/beam/transforms/xlang/inference | Package inference has the cross language implementation of RunInference API implemented in Python SDK. |
| go/pkg/beam/transforms/xlang/python | Package python contains data structures required for python external transforms in a multilanguage pipeline. |
| go/pkg/beam/transforms/xlang/schema | Package schema has the cross language implementation for calling schema transforms in other language SDKs. |
| go/pkg/beam/util | |
| go/pkg/beam/util/diagnostics | Package diagnostics is a beam internal package that contains code for writing and uploading diagnostic info (e.g. |
| go/pkg/beam/util/errorx | Package errorx contains utilities for handling errors. |
| go/pkg/beam/util/execx | Package execx contains wrappers and utilities for the exec package. |
| go/pkg/beam/util/fsx | Package fsx contains utilities for working with filesystems. |
| go/pkg/beam/util/gcsx | Package gcsx contains utilities for working with Google Cloud Storage (GCS). |
| go/pkg/beam/util/grpcx | Package grpcx contains utilities for working with gRPC. |
| go/pkg/beam/util/harnessopts | Package harnessopts defines user-facing entrypoints into Beam hooks affecting the SDK harness. |
| go/pkg/beam/util/pubsubx | Package pubsubx contains utilities for working with Google PubSub. |
| go/pkg/beam/util/shimx | Package shimx specifies the templates for generating type assertion shims for Apache Beam Go SDK pipelines. |
| go/pkg/beam/util/starcgenx | Package starcgenx is a Static Analysis Type Assertion shim and Registration Code Generator which provides an extractor to extract types from a package, in order to generate appropriate shims for a package so code can be generated for it. |
| go/pkg/beam/util/structx | Package structx provides utilities for working with structs. |
| go/pkg/beam/util/syscallx | Package syscallx provides system call utilities that attempt to hide platform differences. |
| go/pkg/beam/x | |
| go/pkg/beam/x/beamx | Package beamx is a convenience package for beam. |
| go/pkg/beam/x/debug | Package debug contains pipeline components that may help in debugging pipeline issues. |
| go/pkg/beam/x/hooks | |
| go/pkg/beam/x/hooks/perf | Package perf is to add performance measuring hooks to a runner, such as cpu, heap, or trace profiles. |
- Version
- v2.66.0 (latest)
- Published
- Jun 23, 2025
- Platform
- linux/amd64
- Last checked
- 7 hours ago –
Tools for package owners.