Skip to content

Data Source Filter Predicate

Filter is the <> for <> that can be pushed down to a relation (aka data source).

Filter is used when:

[[contract]] [source, scala]


package org.apache.spark.sql.sources

abstract class Filter { // only required methods that have no implementation // the others follow def references: Array[String] }


.Filter Contract [cols="1,2",options="header",width="100%"] |=== | Method | Description

| references a| [[references]] Column references, i.e. list of column names that are referenced by a filter

Used when:

  • Filter is requested to <>

  • <>, <> and <> filters are requested for the <> |===

[[implementations]] .Filters [cols="1,2",options="header",width="100%"] |=== | Filter | Description

| And | [[And]]

| EqualNullSafe | [[EqualNullSafe]]

| EqualTo | [[EqualTo]]

| GreaterThan | [[GreaterThan]]

| GreaterThanOrEqual | [[GreaterThanOrEqual]]

| In | [[In]]

| IsNotNull | [[IsNotNull]]

| IsNull | [[IsNull]]

| LessThan | [[LessThan]]

| LessThanOrEqual | [[LessThanOrEqual]]

| Not | [[Not]]

| Or | [[Or]]

| StringContains | [[StringContains]]

| StringEndsWith | [[StringEndsWith]]

| StringStartsWith | [[StringStartsWith]] |===

=== [[findReferences]] Finding Column References in Any Value -- findReferences Method

[source, scala]

findReferences(value: Any): Array[String]

findReferences takes the <> from the value filter is it is one or returns an empty array.

NOTE: findReferences is used when <>, <>, <>, <>, <>, <> and <> filters are requested for their <>.

v2references

v2references: Array[Array[String]]

v2references...FIXME

v2references is used when...FIXME

Back to top