/sap-hana-ml

SAP HANA DataFrame Reference

Module: hana_ml.dataframe Documentation: https://help.sap.com/doc/1d0ebfe5e8dd44d09606814d83308d4b/2.0.07/en-US/hana_ml.dataframe.html

Table of Contents

• ConnectionContext
  • Constructor
  • Connection Management Methods
  • ABAP SQL Operations
  • Schema Operations
  • Table Operations
  • View Operations
  • Procedure Operations
  • Temporary Table Management
  • System Information
  • Data Operations
  • Streaming Operations
  • SQL Execution
• DataFrame
  • Creating DataFrames
  • Properties
  • Data Selection
  • Filtering
  • Data Transformation
  • Sorting
  • Aggregations
  • Join Operations
  • Set Operations
  • Data Output
  • Feature Engineering
  • Data Quality
  • Utility Methods
• Utility Functions
• Lazy Evaluation

ConnectionContext

Manages database connections to SAP HANA instances.

Constructor

from hana_ml import ConnectionContext

conn = ConnectionContext(
    address='hostname',          # HANA host
    port=443,                    # Port (443 for HANA Cloud)
    user='username',             # Database user
    password='password',         # Password
    encrypt=True,                # SSL encryption (required for Cloud)
    sslValidateCertificate=True, # Certificate validation
    autocommit=True              # Auto-commit transactions
)

Connection Management Methods

ABAP SQL Operations

Schema Operations

Table Operations

View Operations

Procedure Operations

Temporary Table Management

System Information

SQL Execution

Data Operations

Streaming Operations

DataFrame

Represents tabular data from HANA with lazy evaluation.

Properties

Data Selection

# Select specific columns
df.select('COL1', 'COL2', 'COL3')

# Deselect columns
df.deselect('UNWANTED_COL')

# Get first/last rows
df.head(10)
df.tail(10)
df.to_head()  # Limit to first N rows
df.to_tail()  # Limit to last N rows

Filtering

# Simple filter
df.filter("AGE > 30")

# Multiple conditions
df.filter("AGE > 30 AND SALARY > 50000")

# Check for value existence
df.has('COLUMN', 'value')

# Remove duplicates
df.distinct()
df.drop_duplicates()

# Handle missing values
df.dropna()                    # Remove rows with NULL
df.dropna(subset=['COL1'])     # Only check specific columns
df.fillna(0)                   # Replace NULL with value
df.fillna({'COL1': 0, 'COL2': 'unknown'})  # Column-specific

Data Transformation

# Add columns
df.add_id('ROW_ID')                    # Add auto-increment ID
df.add_constant('NEW_COL', value=1)    # Add constant column

# Rename
df.alias('new_df_name')                # Rename DataFrame
df.rename_columns({'OLD': 'NEW'})      # Rename columns

# Type conversion
df.cast('COLUMN', 'DOUBLE')            # Cast to specific type
df.auto_cast()                         # Auto-detect types

# Column operations
df.split_column('FULL_NAME', ' ', ['FIRST', 'LAST'])
df.concat_columns(['FIRST', 'LAST'], 'FULL_NAME', delimiter=' ')

# Value replacement
df.nullif('COLUMN', 'NA')              # Replace value with NULL
df.replace('COLUMN', {'old': 'new'})   # Replace values

Sorting

df.sort('COLUMN', desc=True)
df.sort_values(['COL1', 'COL2'], ascending=[True, False])
df.sort_index()

Aggregations

# Basic aggregations
df.count()
df.min('COLUMN')
df.max('COLUMN')
df.sum('COLUMN')
df.mean('COLUMN')
df.median('COLUMN')
df.stddev('COLUMN')

# Custom aggregations
df.agg([
    ('SALARY', 'mean', 'AVG_SALARY'),
    ('SALARY', 'max', 'MAX_SALARY'),
    ('AGE', 'count', 'COUNT')
])

# Correlation
df.corr('COL1', 'COL2')

# Value counts
df.value_counts('CATEGORY')

# Pivot table
df.pivot_table(
    values='SALES',
    index='REGION',
    columns='QUARTER',
    aggfunc='sum'
)

# Descriptive statistics
df.summary()
df.describe()

Join Operations

Join conditions use SQL-style syntax with fully qualified column references:

# Inner join - condition uses SQL-like syntax with DataFrame names
df1.join(df2, condition='df1.ID = df2.ID')  # ✓ Correct: explicit table refs

# Left join
df1.join(df2, condition='df1.ID = df2.ID', how='left')

# Right join
df1.join(df2, condition='df1.ID = df2.ID', how='right')

# Full outer join
df1.join(df2, condition='df1.ID = df2.ID', how='full')

# Join types: 'inner' (default), 'left', 'right', 'full'

Important: Always use fully qualified column names (df1.COLUMN, df2.COLUMN) in join conditions to avoid ambiguity:

# ✓ Correct - explicit DataFrame references
df1.join(df2, condition='df1.ID = df2.CUSTOMER_ID', how='left')

# ✗ Avoid - ambiguous column references may cause errors
# df1.join(df2, condition='ID = CUSTOMER_ID', how='left')

Set Operations

# Union
df1.union(df2)

# Generic set operations
df1.set_operations(df2, operation='EXCEPT')      # Difference
df1.set_operations(df2, operation='INTERSECT')   # Intersection

Data Output

# Materialize to pandas
pdf = df.collect()

# Save to HANA table
df.save('NEW_TABLE', force=True)
df.save_nativedisktable('NATIVE_TABLE')

# Serialize
df.to_pickle('filename.pkl')

# Date conversion
df.to_datetime('DATE_COL', format='%Y-%m-%d')

Feature Engineering

# Binning
df.bin('COLUMN', bins=[0, 25, 50, 75, 100], labels=['Q1', 'Q2', 'Q3', 'Q4'])

# Generate features
df.generate_feature('NEW_COL', 'COL1 + COL2')

# Computed columns
df.mutate('NEW_COL', 'COL1 * COL2')

# Difference calculation
df.diff('COLUMN', periods=1)

Data Quality

# Check for empty
df.empty()

# Check for NULL values
df.hasna()

# Check for constant columns
df.has_constant_columns()
df.drop_constant_columns()

# Column type checking
df.is_numeric('COLUMN')

Utility Methods

# Configuration
df.set_name('new_name')
df.set_index('ID_COLUMN')
df.set_source_table('ORIGINAL_TABLE')

# Column reordering
df.rearrange(['COL3', 'COL1', 'COL2'])

# Schema inspection
df.get_table_structure()

# Drop columns
df.drop('UNWANTED_COL')

# Validation
df.enable_validate_columns()
df.disable_validate_columns()

# Type definitions
df.generate_table_type()
df.declare_lttab_usage()

Utility Functions

DataFrame Creation

from hana_ml.dataframe import (
    create_dataframe_from_pandas,
    create_dataframe_from_spark,
    create_dataframe_from_shapefile
)

# From pandas
hdf = create_dataframe_from_pandas(
    conn,
    pandas_df,
    table_name='MY_TABLE',
    schema='MY_SCHEMA',
    force=True  # Overwrite if exists
)

# From Spark
hdf = create_dataframe_from_spark(conn, spark_df, table_name='MY_TABLE')

# From shapefile (geospatial)
hdf = create_dataframe_from_shapefile(conn, 'path/to/file.shp', table_name='GEO_TABLE')

Data Import

from hana_ml.dataframe import import_csv_from

# Import CSV
hdf = import_csv_from(
    conn,
    'path/to/file.csv',
    table_name='CSV_TABLE',
    delimiter=',',
    header=True
)

Serialization

from hana_ml.dataframe import read_pickle

# Read pickled DataFrame
hdf = read_pickle(conn, 'filename.pkl')

Reshaping

from hana_ml.dataframe import melt

# Unpivot DataFrame
melted = melt(
    df,
    id_vars=['ID', 'NAME'],
    value_vars=['Q1', 'Q2', 'Q3', 'Q4'],
    var_name='QUARTER',
    value_name='SALES'
)

SQL Utilities

from hana_ml.dataframe import quotename

# Safe SQL identifier quoting
safe_name = quotename('table-with-special-chars')

Lazy Evaluation

HANA DataFrames use lazy evaluation - operations build SQL expressions without immediate execution.

# These operations build SQL but don't execute
filtered = df.filter("AGE > 30")
selected = filtered.select('NAME', 'AGE')
sorted_df = selected.sort('AGE', desc=True)

# Only collect() triggers execution
result = sorted_df.collect()  # Returns pandas DataFrame

Benefits:

• Query optimization across chained operations
• Reduced data transfer
• Push computations to database