Jeremy Monat, PhD

Prioritizing Drug-Like ChEMBL Compounds Within Target Profiles

2026-01-05T00:00:00+00:00

When reviewing data to find pharma compounds for virtual screening, we might want to check what their target profiles and rank candidates by how many Lipinski’s rule of five violations they have–the fewer the better. Here, a target profile refers to the set of targets a compound is known to be active against. This post uses the ChEMBL API and a SQLite database to do that.

This post pulls data from ChEMBL using its chembl_webresource_client for Python. It’s a helpful package which handles the ChEMBL API calls. It also provides caching so you won’t accidentally run the same queries more than once. APIs often ask users to cache the results; I like that ChEMBL goes ahead and does that for you. (If it didn’t, I would have used DiskCache, which as the name implies caches results to disk so they persist across code runs, and which I’ve found works well for storing results from other API calls.) I also like that the package handles obeying the API rate limit rather than setting a limit in units which may be difficult to measure and then penalizing you for exceeding it.

We write the results directly to a SQLite database. SQLite is file-based so its uptime is nearly 100% as long as your code is running on the same system. That means we don’t need to worry about its availability. Of course it being file-based is not ideal if users are distributed across the Internet, but that’s not what we’re doing here.

Open this notebook in marimo molab or Google Colab so you can run it without installing anything on your computer.

Virtual environment setup with uv

To recreate the environment for this notebook, install it with uv:

uv sync

Database schemas—conceptual

I find database entity-relationship diagrams to be useful for planning and documenting.

ChEMBL has the connections as compounds ↔ activities ↔ targets. Let’s plot that as an ERD using graphviz.

import logging
import time
from collections import defaultdict

import pydot
import sqlalchemy
from sqlalchemy import (
    Column,
    Float,
    Integer,
    String,
    create_engine,
    func,
    select,
    UniqueConstraint,
)
from sqlalchemy.dialects.sqlite import insert
from sqlalchemy.exc import IntegrityError, SQLAlchemyError
from sqlalchemy.orm import DeclarativeBase, sessionmaker
from chembl_webresource_client.new_client import new_client
from graphviz import Digraph
from IPython.display import SVG, display
from sqlalchemy_schemadisplay import create_schema_graph
from rdkit import Chem
from rdkit.Chem import MolFromSmiles
from rdkit.Chem.Draw import MolsMatrixToGridImage

We’ll set some parameters for ERDs in graphviz.

# Scale down size of ERD relationship labels and arrowheads
crow_fontsize = "8"  # smaller font for labels
arrow_scale = "0.4"  # smaller arrowheads

# Create the diagram
dot_chembl = Digraph(format="svg")
dot_chembl.attr(rankdir="LR", splines="ortho")
dot_chembl.attr(
    "node", shape="record", fontsize="10", style="filled", fillcolor="lightgrey"
)

# Nodes
dot_chembl.node("Compound", "{Compound|compound_id (PK)}", fillcolor="#A3C1DA")
dot_chembl.node("Activity", "{Activity|activity_id (PK)}", fillcolor="#A3C1DA")
dot_chembl.node("Target", "{Target|target_id (PK)}", fillcolor="#A3C1DA")

# Relationships
dot_chembl.edge(
    "Compound",
    "Activity",
    fontsize=crow_fontsize,
    arrowhead="crow",
    arrowtail="crow",
    dir="both",
    color="black",
    arrowsize=arrow_scale,
)
dot_chembl.edge(
    "Activity",
    "Target",
    fontsize=crow_fontsize,
    arrowhead="crow",
    arrowtail="crow",
    dir="both",
    color="black",
    arrowsize=arrow_scale,
)

# Render inline SVG
svg_chembl = dot_chembl.pipe(format="svg").decode("utf-8")
display(SVG(svg_chembl))

That makes sense as a comprehensive schema; in this code, I wanted to simplify it by connecting compounds to targets directly. So my schema is simply compounds ↔ targets:

# Create the diagram
dot_simple = Digraph(format="svg")
dot_simple.attr(rankdir="LR", splines="ortho")
dot_simple.attr(
    "node", shape="record", fontsize="10", style="filled", fillcolor="lightgrey"
)

# Define nodes
dot_simple.node("Compound", "{Compound|compound_id (PK)}", fillcolor="#A3C1DA")
dot_simple.node("Target", "{Target|target_id (PK)}", fillcolor="#A3C1DA")

# Many-to-many edge (crow's foot at both ends)
dot_simple.edge(
    "Compound",
    "Target",
    fontsize=crow_fontsize,
    arrowhead="crow",
    arrowtail="crow",
    dir="both",
    color="black",
    arrowsize=arrow_scale,
)

# Render inline SVG
svg_simple = dot_simple.pipe(format="svg").decode("utf-8")
display(SVG(svg_simple))

Code setup

# Set logger level to INFO
logging.basicConfig(
    level=logging.INFO,
    format="[%(levelname)s] %(message)s",
    force=True,
)
logger = logging.getLogger(__name__)

Fetching data from ChEMBL

Now let’s get the data from ChEMBL. The chembl_webresource_client new_client provides objects for molecule, activity, and target.

We’ll start with a set of molecules, then fetch associated data. Here we fetch a number of compounds n_compounds starting at ChEMBL ID start_id. Because ChEMBL assigns a ChEMBL ID to not only compounds but also targets, we’ll count how many IDs correspond to compounds.

Then we’ll fetch associated activities (interactions between compounds and targets) for those compounds. Let’s say we’re interested in older (before 2010) studies on humans with IC50 standards and binding assays. We specify those as filters when we ask for the activities. This will serve to limit us to a reasonable number of activities and thus targets.

Then we get target data including its ChEMBL ID, name, type, and organism.

Lastly we associate targets with compounds by creating a list of targets (the compound’s target profile) for each molecule. This will make it easier to populate our database tables.

def get_chembl_molecules(
    n_compounds: int = 2,
    start_id: int = 1,
):
    # Create list of ChEMBL ids (e.g. CHEMBL12)
    chembl_ids = [f"CHEMBL{id}" for id in range(start_id, start_id + n_compounds)]

    molecule = new_client.molecule
    activity = new_client.activity
    target = new_client.target

    # Suppress INFO logs from the chembl package while keeping global INFO level
    logging.getLogger("chembl_webresource_client").setLevel(logging.WARNING)

    # --------------------
    # 1) Fetch molecules
    # --------------------
    mols = list(
        molecule.filter(molecule_chembl_id__in=chembl_ids).only(
            [
                "molecule_chembl_id",
                "molecule_structures",
                "pref_name",
                "molecule_properties",
            ]
        )
    )

    # From the molecules in mols, extract the digits after "CHEMBL" to check which IDs were found
    chembl_ids_found = set()
    for mol in mols:
        chembl_id = mol.get("molecule_chembl_id", "")
        if chembl_id.startswith("CHEMBL"):
            chembl_ids_found.add(int(chembl_id.replace("CHEMBL", "")))
    chembl_ids_found_str = ", ".join(map(str, sorted(chembl_ids_found)))

    logger.info(
        f"Of the {n_compounds} ChEMBL IDs "
        f"({start_id}-{start_id + n_compounds - 1}), {len(mols)} are compounds: "
        f"ChEMBL IDs {chembl_ids_found_str}"
    )

    if not mols:
        return []

    mol_ids_present = [m["molecule_chembl_id"] for m in mols]

    # ---------------------------------
    # 2) Bulk fetch activities → targets
    # ---------------------------------
    activities = activity.filter(
        molecule_chembl_id__in=mol_ids_present,
        target_organism="Homo sapiens",
        standard_type="IC50",
        assay_type="B",
        document_year__lt=2010,
    ).only(
        [
            "molecule_chembl_id",
            "target_chembl_id",
        ]
    )
    logger.info(f"Fetched {len(activities)} activities from ChEMBL.")

    mol_to_target_ids = defaultdict(set)

    for act in activities:
        try:
            mol_to_target_ids[act["molecule_chembl_id"]].add(
                act["target_chembl_id"]
            )
        except Exception as e:
            logger.warning(e)

    # ---------------------------------
    # 3) Fetch target metadata (bulk)
    # ---------------------------------
    all_target_ids = sorted(
        {tar_id for tar_ids in mol_to_target_ids.values() for tar_id in tar_ids}
    )

    # Create dictionary of ChEMLB ID: target entries
    targets = {}
    if all_target_ids:
        for t in target.filter(target_chembl_id__in=all_target_ids).only(
            [
                "target_chembl_id",
                "pref_name",
                "target_type",
                "organism",
            ]
        ):
            targets[t["target_chembl_id"]] = t
    logger.info(f"Fetched metadata for {len(targets)} targets from ChEMBL.")

    # ---------------------------------
    # 4) Attach targets to molecules
    # ---------------------------------

    for m in mols:
        t_ids = mol_to_target_ids.get(m["molecule_chembl_id"], [])
        m["targets"] = [targets[tar_id] for tar_id in t_ids if tar_id in targets]

    return mols, all_target_ids

SQLite database

While we use a SQLite database, we interact with it using SQLAlchemy. SQLAlchemy has the advantage that the code is the same regardless of which database type you use (except for some database-specific idioms), so if you decide to change database type later, you don’t have to rewrite the code that interacts with the database.

Setting up the database

Here we set up the Base and the SQLite database name (compounds.db), then define a Session.

engine = create_engine("sqlite:///compounds.db", echo=False)
Session = sessionmaker(bind=engine)

class Base(DeclarativeBase):
    pass

Now we create a class for each table. In addition to having a database-assigned primary key id, we enforce uniqueness on the ChEMBL ID to make sure we don’t add the same compound or target multiple times to its table.

The final table is a join table between the compound and target tables. We set a uniqueness constraint to ensure that each compound-target pair can be added only once.

class Compound(Base):
    __tablename__ = "compound"

    id = Column(Integer, primary_key=True)
    chembl_id = Column(Integer, unique=True)
    sml = Column(String)
    pref_name = Column(String)
    molwt = Column(Float)  # MolWt
    tpsa = Column(Float)  # TPSA
    num_h_acceptors = Column(Integer)  # NumHAcceptors
    num_h_donors = Column(Integer)  # NumHDonors
    num_ro5 = Column(Integer)  # NumRo5
    mol_logp = Column(Float)  # MolLogP

class Target(Base):
    __tablename__ = "target"

    id = Column(Integer, primary_key=True)
    target_chembl_id = Column(String, unique=True)
    organism = Column(String)
    pref_name = Column(String)
    target_type = Column(String)

class CompoundTarget(Base):
    __tablename__ = "compound_target"

    id = Column(Integer, primary_key=True)
    compound_id = Column(Integer, sqlalchemy.ForeignKey("compound.id"))
    target_id = Column(Integer, sqlalchemy.ForeignKey("target.id"))

    __table_args__ = (
        UniqueConstraint("compound_id", "target_id", name="uq_compound_target"),
    )

Now we define simple functions to reset and initialize the database. These are one-liners so they’re set up as functions, and named, to remind us of what they do.

def reset_db():
    """Drop all database tables (use with caution)."""
    Base.metadata.drop_all(engine)

def init_db():
    """Create database tables (call once at app startup or from scripts/tests)."""
    Base.metadata.create_all(engine)

Visualizing the database schema

Now that we’ve created the database in code, let’s visualize it to make sure it’s as we planned.

def add_ordering_edges(graph, Base, exclude_tables=None):
    """
    Add invisible edges based on foreign key relationships to enforce left-to-right ordering

    Args:
        graph: A pydot.Dot graph object (e.g. ERD) to add edges to.
        Base: SQLAlchemy declarative base containing table metadata.
        exclude_tables: Set of table names to exclude from processing

    Returns:
        The modified graph object with invisible ordering edges added.
    """
    if exclude_tables is None:
        exclude_tables = set()

    # Get all table names
    tables = Base.metadata.tables.keys()

    # For each table, check for foreign keys
    for table_name in tables:
        if table_name in exclude_tables:
            continue

        table = Base.metadata.tables[table_name]

        for fk in table.foreign_key_constraints:
            parent_table = fk.referred_table.name

            if parent_table not in exclude_tables:
                # Add invisible edge: parent -> child
                graph.add_edge(pydot.Edge(parent_table, table_name, style="invis"))

    return graph

# Create the ERD graph
graph_full = create_schema_graph(
    engine=engine,
    metadata=Base.metadata,
    show_datatypes=True,
    show_indexes=False,
    rankdir="LR",
    concentrate=False,
)

# Force strict left-to-right ordering with invisible edges;
# add them programmatically by inspecting the SQLAlchemy model
add_ordering_edges(graph_full, Base)

graph_full.set("splines", "ortho")

# Move FK labels horizontally away from edges
for edge_full in graph_full.get_edges():
    head_full = edge_full.get_headlabel()
    tail_full = edge_full.get_taillabel()

    if head_full:
        clean_head_full = head_full.replace("+ ", "").replace("+", "")
        edge_full.set_headlabel(clean_head_full)

    if tail_full:
        clean_tail_full = tail_full.replace("+ ", "").replace("+", "")
        edge_full.set_taillabel(clean_tail_full)

        edge_full.set_label("")  # critical fix
        edge_full.set_labeldistance("2.5")

# Increase horizontal spacing between tables
graph_full.set("ranksep", "1.0")
svg_content_full = graph_full.create_svg()
display(SVG(svg_content_full))

Let’s simplify the ERD by showing the relationship between the compound and target tables as a many-to-many relationship. We can do that programmatically by detecting the join table, removing it, and replacing it with a (conceptual) many-to-many relationship between the two remaining tables.

# ERD utilities
def detect_join_tables(Base):
    """
    Detect join tables (tables with only an id and two foreign keys).

    Returns:
        dict: Mapping of join_table_name -> (parent_table1, parent_table2)
    """
    join_tables = {}

    for table_name, table in Base.metadata.tables.items():
        # Get foreign key constraints
        fk_constraints = list(table.foreign_key_constraints)

        # A join table typically has:
        # - Only id as primary key (or composite PK of the two FKs)
        # - Exactly 2 foreign key columns
        # - Minimal or no other columns

        foreign_key_columns = []
        for fk in fk_constraints:
            foreign_key_columns.extend(fk.column_keys)

        # Check if this looks like a join table:
        # Has exactly 2 FK constraints pointing to different tables
        if len(fk_constraints) == 2:
            referred_tables = [fk.referred_table.name for fk in fk_constraints]

            # Make sure they point to different tables
            if referred_tables[0] != referred_tables[1]:
                join_tables[table_name] = tuple(referred_tables)

    return join_tables

def get_primary_key_name(Base, table_name):
    """
    Get the primary key column name for a table.

    Args:
        Base: SQLAlchemy declarative base
        table_name: Name of the table

    Returns:
        str: Primary key column name (or comma-separated list if composite)
    """
    table = Base.metadata.tables[table_name]
    pk_columns = [col.name for col in table.columns if col.primary_key]

    if pk_columns:
        return ", ".join(pk_columns)
    else:
        # Fallback
        return "id"

def remove_join_tables_from_graph(graph, Base):
    """
    Remove join tables from graph and replace with direct many-to-many edges.

    Args:
        graph: pydot.Dot graph object
        Base: SQLAlchemy declarative base

    Returns:
        set: Names of removed join tables
    """
    join_tables = detect_join_tables(Base)

    for join_table, (table1, table2) in join_tables.items():
        # Remove the join table node
        graph.del_node(join_table)

        # Remove all edges connected to the join table
        for edge in list(graph.get_edges()):
            if (
                edge.get_source() == join_table
                or edge.get_destination() == join_table
            ):
                graph.del_edge(edge.get_source(), edge.get_destination())

        # Get primary key names for both tables
        table1_pk = get_primary_key_name(Base, table1)
        table2_pk = get_primary_key_name(Base, table2)

        # Add a direct many-to-many edge between the two tables
        graph.add_edge(
            pydot.Edge(
                table1,
                table2,
                taillabel=table1_pk,
                headlabel=table2_pk,
                arrowhead="crow",
                arrowtail="crow",
                dir="both",
            )
        )

    return set(join_tables.keys())

Now we can actually simplify the ERD.

# Create the simplified ERD graph
graph = create_schema_graph(
    engine=engine,
    metadata=Base.metadata,
    show_datatypes=True,
    show_indexes=False,
    rankdir="LR",
    concentrate=False,
)

# Automatically detect and remove join tables
excluded_tables = remove_join_tables_from_graph(graph, Base)

# Add ordering edges (excluding detected join tables)
add_ordering_edges(graph, Base, exclude_tables=excluded_tables)

graph_full.set("splines", "ortho")

# Move FK labels horizontally away from edges
for edge in graph.get_edges():
    head = edge.get_headlabel()
    tail = edge.get_taillabel()

    if head:
        edge.set_headlabel(head)

    if tail:
        edge.set_taillabel(tail)
        edge.set_labeldistance("2.5")

graph_full.set("ranksep", "1.0")

svg_content = graph.create_svg()
display(SVG(svg_content))

Saving data to SQLite database

Now let’s define a function to save our compounds and targets to our SQLite database. To avoid duplication, we start by preloading all the targets into that table and returning the ChEMBL and database ids. The trick is the “returning” part, .returning(Target.target_chembl_id, Target.id). That lets us create a dictionary mapping our input data (ChEMBL ID which we already had) to our database id (which was just created), which is an O(1) (constant time) lookup so we can quickly link the compound to the target. That saves us from having to query the database each time we want to associate a target with a compound.

We do the same for compounds, adding them in bulk, returning their ChEMBL and database ids, and creating a dictionary.

After that, we have the database IDs for both compounds and targets, allowing us to create compound-target records quickly in memory and again bulk adding them to the database without querying the database for the compound or target IDs.

Note that creating the dictionaries does require some RAM, so if you were creating a huge number of records, memory might become a limitation.

def save_compounds_to_db(
    molecules: list[dict], all_target_ids
) -> tuple[int, int, int]:
    """Save multiple compounds and their targets to the database efficiently using bulk inserts."""

    # Deduplicate targets across all molecules
    target_map: dict[str, dict] = {
        t["target_chembl_id"]: {
            "organism": t.get("organism"),
            "pref_name": t.get("pref_name"),
            "target_chembl_id": t.get("target_chembl_id"),
            "target_type": t.get("target_type"),
        }
        for m in molecules
        for t in m.get("targets", [])
        if t.get("target_chembl_id")
    }
    all_targets = list(target_map.values())

    # Build compound records for bulk insert, deduplicating compounds by chembl_id
    compound_map_input: dict[str, dict] = {}
    for mol in molecules:
        chembl_id = mol.get("molecule_chembl_id")
        if not chembl_id or chembl_id in compound_map_input:
            continue
        props = mol.get("molecule_properties") or {}
        compound_map_input[chembl_id] = {
            "chembl_id": chembl_id,
            "sml": (mol.get("molecule_structures") or {}).get("canonical_smiles"),
            "pref_name": mol.get("pref_name"),
            "molwt": props.get("full_molweight"),
            "tpsa": props.get("tpsa"),
            "num_h_acceptors": props.get("num_h_acceptors"),
            "num_h_donors": props.get("num_h_donors"),
            "num_ro5": props.get("num_ro5_violations"),
            "mol_logp": props.get("alogp"),
        }

    compound_records = list(compound_map_input.values())

    n_targets_saved = 0
    n_mols_saved = 0
    n_compounds_targets_saved = 0

    with Session() as db_session:
        try:
            # Bulk insert targets, get back chembl_id -> db id mapping
            existing_targets: dict[str, int] = {}
            if all_targets:
                result = db_session.execute(
                    insert(Target)
                    .on_conflict_do_nothing(index_elements=["target_chembl_id"])
                    .returning(Target.target_chembl_id, Target.id),
                    all_targets,
                )
                existing_targets = {row.target_chembl_id: row.id for row in result}

                # Fetch any pre-existing targets that were skipped by on_conflict_do_nothing
                all_target_chembl_ids = [t["target_chembl_id"] for t in all_targets]
                missing_target_ids = [
                    tid
                    for tid in all_target_chembl_ids
                    if tid not in existing_targets
                ]
                if missing_target_ids:
                    rows = db_session.execute(
                        select(Target.target_chembl_id, Target.id).where(
                            Target.target_chembl_id.in_(missing_target_ids)
                        )
                    )
                    existing_targets.update(
                        {row.target_chembl_id: row.id for row in rows}
                    )
                n_targets_saved = len(existing_targets)

            # Bulk insert compounds, get back chembl_id -> db id mapping
            result = db_session.execute(
                insert(Compound)
                .on_conflict_do_nothing(index_elements=["chembl_id"])
                .returning(Compound.chembl_id, Compound.id),
                compound_records,
            )
            compound_map = {row.chembl_id: row.id for row in result}

            # Fetch any pre-existing compounds that were skipped by on_conflict_do_nothing
            all_chembl_ids = [c["chembl_id"] for c in compound_records]
            missing_compound_ids = [
                cid for cid in all_chembl_ids if cid not in compound_map
            ]
            if missing_compound_ids:
                rows = db_session.execute(
                    select(Compound.chembl_id, Compound.id).where(
                        Compound.chembl_id.in_(missing_compound_ids)
                    )
                )
                compound_map.update({row.chembl_id: row.id for row in rows})
            n_mols_saved = len(compound_map)

            # Build all CompoundTarget join rows in memory
            compound_target_records = []
            seen_pairs = set()
            for mol in molecules:
                chembl_id = mol.get("molecule_chembl_id")
                compound_id = compound_map.get(chembl_id)
                if not compound_id:
                    logger.warning(
                        f"No DB id found for compound {chembl_id}, skipping its targets"
                    )
                    continue

                for target_data in mol.get("targets", []):
                    target_chembl_id = target_data.get("target_chembl_id")
                    target_id = existing_targets.get(target_chembl_id)
                    if not target_id:
                        logger.warning(
                            f"No DB id found for target {target_chembl_id}, skipping"
                        )
                        continue
                    pair = (compound_id, target_id)
                    if pair in seen_pairs:
                        continue
                    seen_pairs.add(pair)
                    compound_target_records.append(
                        {"compound_id": compound_id, "target_id": target_id}
                    )

            # Bulk insert all CompoundTarget join rows
            if compound_target_records:
                db_session.execute(
                    insert(CompoundTarget).on_conflict_do_nothing(
                        index_elements=["compound_id", "target_id"]
                    ),
                    compound_target_records,
                )
                n_compounds_targets_saved = len(compound_target_records)

            db_session.commit()

        except IntegrityError as e:
            logger.info(f"IntegrityError while saving: {e}")
            db_session.rollback()
        except SQLAlchemyError as e:
            logger.exception(f"Database error saving compounds: {e}")
            db_session.rollback()
            raise

    return n_mols_saved, n_targets_saved, n_compounds_targets_saved

Let’s go ahead and set up the database.

# Reset database
reset_db()

# Ensure tables exist
init_db()

Now let’s actually get the molecules from ChEMBL and save them to our SQLite database.

# Measure how long it takes to fetch ChEMBL molecules
start = time.time()
mols, all_target_ids = get_chembl_molecules(
    start_id=796,
    n_compounds=14,
)

end = time.time()
logger.info(
    f"Fetched {len(mols)} molecules and associated activities in {end - start:.2f} seconds from ChEMBL."
)

start = time.time()

n_mols_saved, n_targets_saved, n_compounds_targets_saved = save_compounds_to_db(
    mols, all_target_ids
)
logger.info(
    f"Saved {n_mols_saved} compounds, "
    f"{n_targets_saved} targets, "
    f"and {n_compounds_targets_saved} compound-target associations to the database, "
    f"in {time.time() - start:.2f} seconds."
)

[INFO] Of the 14 ChEMBL IDs (796-809), 12 are compounds: ChEMBL IDs 796, 797, 799, 800, 801, 802, 803, 804, 806, 807, 808, 809
[INFO] Fetched 26 activities from ChEMBL.
[INFO] Fetched metadata for 16 targets from ChEMBL.
[INFO] Fetched 12 molecules and associated activities in 0.01 seconds from ChEMBL.
[INFO] Saved 12 compounds, 16 targets, and 19 compound-target associations to the database, in 0.00 seconds.

Getting compound-target results

Now we can get the results we’re interested in.

Grouping compounds by target profiles

First we’ll simply group the compounds by target profiles. We list the compounds for each set of targets, ordering the compounds within a target profile by ChEMBL ID.

with Session() as db_session1:
    # For each compound, concatenate its targets with a slash.
    # Then, count how many distinct such tuples exist in the database and list them in order of ChEMBL ID.

    # Build per-compound target profile subquery:
    # as correlated scalar subquery: group_concat over an ordered selection of types to ensure consistent ordering.

    # correlated inner select returning pref_name for the current Compound, ordered
    inner = (
        select(Target.pref_name)
        .select_from(
            Target.__table__.join(
                CompoundTarget.__table__, CompoundTarget.target_id == Target.id
            )
        )
        .where(CompoundTarget.compound_id == Compound.id)
        .order_by(func.lower(Target.pref_name))
        .correlate(Compound)
    )

    # name the derived table so the outer group_concat can select from it
    ordered_targets = inner.subquery("ordered_targets")

    # aggregate the ordered names with a '\' separator
    target_profile_subq = select(
        func.group_concat(ordered_targets.c.pref_name, "\\")
    ).scalar_subquery()

    # Create a subquery that selects each compound's id and its target profile.
    target_profiles = db_session1.query(
        Compound.id.label("compound_id"),
        target_profile_subq.label("target_profile"),
    ).subquery()

    # Ensure compounds will be ordered by their ChEMBL ID
    subq = (
        db_session1.query(
            target_profiles.c.target_profile,
            Compound.chembl_id,
        )
        .join(Compound, Compound.id == target_profiles.c.compound_id)
        .order_by(Compound.chembl_id)
    ).subquery()

    # Create a list of distinct type combinations and their counts
    # where each is a tuple like (target profile, # compounds, compound ChEMBL IDs)
    compound_targets = (
        db_session1.query(
            subq.c.target_profile,
            func.count().label("num_compounds"),
            func.group_concat(subq.c.chembl_id, ", ").label("chembl_ids"),
        )
        .group_by(subq.c.target_profile)
        .order_by(subq.c.target_profile)
        .all()
    )

    # Print out the results
    n_compound_by_target = 0
    logger.info("1. Distinct compound target profiles and their counts:")
    for target_profile, count, chembl_ids in compound_targets:
        logger.info(f"    {target_profile}: {count} (Compounds: {chembl_ids})")
        n_compound_by_target += count
    logger.info(
        f"    Total compounds counted by target profiles: {n_compound_by_target}"
    )

[INFO] 1. Distinct compound target profiles and their counts:
[INFO]     None: 4 (Compounds: CHEMBL797, CHEMBL801, CHEMBL802, CHEMBL804)
[INFO]     Androgen receptor\Steroid 17-alpha-hydroxylase/17,20 lyase: 1 (Compounds: CHEMBL806)
[INFO]     Glutamate NMDA receptor; GRIN1/GRIN2B\Solute carrier family 22 member 1: 1 (Compounds: CHEMBL807)
[INFO]     Lanosterol 14-alpha demethylase\Malate dehydrogenase, cytoplasmic: 1 (Compounds: CHEMBL808)
[INFO]     Phosphodiesterase 1\Phosphodiesterase 3: 1 (Compounds: CHEMBL799)
[INFO]     Prostaglandin G/H synthase 1\Prostaglandin G/H synthase 2: 1 (Compounds: CHEMBL800)
[INFO]     Proto-oncogene tyrosine-protein kinase Src\Thymidine kinase 2, mitochondrial\Thymidine kinase, cytosolic: 1 (Compounds: CHEMBL803)
[INFO]     Sodium-dependent dopamine transporter\Sodium-dependent noradrenaline transporter\Sodium-dependent serotonin transporter: 2 (Compounds: CHEMBL796, CHEMBL809)
[INFO]     Total compounds counted by target profiles: 12

For example, a set with multiple targets is listed as

Sodium-dependent dopamine transporter\Sodium-dependent noradrenaline transporter\Sodium-dependent serotonin transporter

I used the backslash \ as the delimiter between a target because a target name can contain other commonly-used delimiters such as as forward slashes /, commas ,, and semicolons ;.

By the way, these three targets are closely-related monoamine transporters which regulate concentrations of extracellular monoamine neurotransmitters and are associated with mental health conditions such as Parkinson’s, ADHD, and depression.

Ranking compounds by rule of 5 violations

Now let’s do an initial ranking of compounds within each target set by how many Lipinski’s rule of five violations they have—the fewer the better. We’ll hide the compounds that don’t have any associated targets.

with Session() as db_session2:
    # Query compounds grouped by type and ordered by ascending number of Rule of 5 violations
    compounds_by_ro5 = (
        db_session2.query(
            target_profiles.c.target_profile,
            Compound.chembl_id,
            Compound.pref_name,
            Compound.num_ro5,
            Compound.sml,
        )
        .join(Compound, Compound.id == target_profiles.c.compound_id)
        .order_by(
            target_profiles.c.target_profile,
            Compound.num_ro5,
        )
        .all()
    )
    logger.info(
        "2. Compounds grouped by target profile and ordered by descending number of Rule of 5 violations:"
    )
    logger.info("        Rule of 5 violation count")
    current_target_profile_ro5 = ""
    for (
        target_profile_ro5,
        chembl_id_ro5,
        pref_name_ro5,
        num_ro5,
        sml_ro5,
    ) in compounds_by_ro5:
        if target_profile_ro5 is None:
            continue
        if target_profile_ro5 != current_target_profile_ro5:
            current_target_profile_ro5 = target_profile_ro5
            logger.info(f"    target profile: {current_target_profile_ro5}")
        logger.info(
            f"        {num_ro5} for {pref_name_ro5.casefold() if pref_name_ro5 else ''} ({chembl_id_ro5})"
        )

[INFO] 2. Compounds grouped by target profile and ordered by descending number of Rule of 5 violations:
[INFO]         Rule of 5 violation count
[INFO]     target profile: Androgen receptor\Steroid 17-alpha-hydroxylase/17,20 lyase
[INFO]         0 for flutamide (CHEMBL806)
[INFO]     target profile: Glutamate NMDA receptor; GRIN1/GRIN2B\Solute carrier family 22 member 1
[INFO]         0 for memantine (CHEMBL807)
[INFO]     target profile: Lanosterol 14-alpha demethylase\Malate dehydrogenase, cytoplasmic
[INFO]         1 for econazole (CHEMBL808)
[INFO]     target profile: Phosphodiesterase 1\Phosphodiesterase 3
[INFO]         0 for cilostazol (CHEMBL799)
[INFO]     target profile: Prostaglandin G/H synthase 1\Prostaglandin G/H synthase 2
[INFO]         0 for  (CHEMBL800)
[INFO]     target profile: Proto-oncogene tyrosine-protein kinase Src\Thymidine kinase 2, mitochondrial\Thymidine kinase, cytosolic
[INFO]         0 for cytarabine (CHEMBL803)
[INFO]     target profile: Sodium-dependent dopamine transporter\Sodium-dependent noradrenaline transporter\Sodium-dependent serotonin transporter
[INFO]         0 for methylphenidate (CHEMBL796)
[INFO]         1 for sertraline (CHEMBL809)

So for example if we’re interested in the target set Sodium-dependent dopamine transporter\Sodium-dependent noradrenaline transporter\Sodium-dependent serotonin transporter, we’d probably consider methylphenidate because it has zero Rule of 5 violations before sertraline that has one violation.

Visualizing the results for small-molecule compounds

If you’ve read my blog you can guess I can’t resist showing these small-molecule compounds. Let’s use my RDKit contribution MolsMatrixToGridImage to show the compounds where

each row is a target profile
each column is a compound for that target profile–MolsMatrixToGridImage is useful because there can be a variable number of compounds per target profile.

# Group compounds by target_profile
grouped = defaultdict(list)
for target_profile_b, _, pref_name_b, num_ro5_b, sml_b in compounds_by_ro5:
    if target_profile_b is None:
        continue
    grouped[target_profile_b].append((pref_name_b, num_ro5_b, sml_b))

# Build matrix of mols and legends
mols_matrix = []
legends_matrix = []
blank_mol = Chem.MolFromSmiles("*")

for target_profile_b, compounds in grouped.items():
    # Blank cell for target profile: Blank molecule
    mol_row = [blank_mol]
    # Blank cell legend: Target profile where each target is on its own line
    legend_row = [target_profile_b.replace("\\", "\n")]

    for pref_name_b, num_ro5_b, sml_b in compounds:
        mol = MolFromSmiles(sml_b) if sml_b else None
        mol_row.append(mol)
        legend = f"{pref_name_b or 'unnamed'} ({num_ro5_b} violations)"
        legend_row.append(legend.casefold())

    mols_matrix.append(mol_row)
    legends_matrix.append(legend_row)

# Generate grid image
MolsMatrixToGridImage(
    molsMatrix=mols_matrix,
    legendsMatrix=legends_matrix,
    subImgSize=(300, 300),
)

# /// script
# dependencies = [
#     "aiohttp>=3.13.2",
#     "chembl-webresource-client>=0.10.9",
#     "graphviz>=0.21",
#     "ipykernel>=6.29.0",
#     "ipython>=9.10.0",
#     "marimo>=0.19.10",
#     "matplotlib>=3.10.8",
#     "nbconvert>=7.16.6",
#     "nbformat>=5.10.4",
#     "pydot>=4.0.1",
#     "rdkit>=2025.9.5",
#     "ruff>=0.15.1",
#     "sqlalchemy>=2.0.45",
#     "sqlalchemy-schemadisplay>=2.0",
# ]
# ///

Hackathon Tips

2025-11-09T00:00:00+00:00

I participated in the LLM Hackathon for Applications in Materials Science & Chemistry September 11-12, 2025. It was a great opportunity to think and learn about how LLMs can be used in science. It was also a chance to meet new people in the community, or collaborate live with folks I know from online. Here are some things I learned about how to get the most done in a hackathon.

I was on the ChemGraph-IR team whose goal was to calculate the infrared absorption spectrum of a molecule given an LLM prompt requesting the IR spectrum of a named chemical.

Infrared absorption spectrum of ethanol generated by Murat Keçeli

Begin with the end in mind®

As this trademarked phrase from Stephen Covey’s The 7 Habits of Highly Effective People® says, it’s a good idea to think from the beginning about what you’ll need to produce at the end of the hackathon.

Plan the deliverables

When a team starts a hackathon, it’s natural to focus first on how to start: What permissions to set up, what code to write, etc. But it’s actually a good idea to think first about what the deliverables will be. For example, for this hackathon the deliverables were a two-minute video, a submission form, and a social media post announcing the completed project.

An effective way to plan a video is with storyboards, that is the still images that provide the flow of the video. So what I would do next hackathon is start by deciding the key assets–project outputs such as graphs. That would give everyone a good starting point:

The software developers would know what they’re working towards, for example a graph of infrared absorption intensity against frequency (cm^-1).
The video creators would know what the storyboards are so they could plan the video. It would probably have alerted the team to the amount of work that needed to go into the video. At least two of us spent most of the second day working on the video. While our ChemGraph video was perfectly fine–it was like a scientific presentation with slides and narration–other teams’ were more elaborate and were projects in themselves. Had someone focused on the presentation from the beginning, they might have come up with an outside-the-box idea.

# Source - https://stackoverflow.com/questions/18019477/how-can-i-play-a-local-video-in-my-ipython-notebook#18026076
# Posted by Viktor Kerkez, modified by community. See post 'Timeline' for change history
# Retrieved 2025-11-06, License - CC BY-SA 4.0

from IPython.display import Video
Video("/images/MethanolVibrations.mov")

Video of infrared absorption spectrum and vibrational modes of methanol made by Kevin Shen

Draft the final texts ahead of time

Our team ended up submitting the form just minutes before the deadline. Anticipating there might be a last-minute crunch, I drafted in a spreadsheet the form responses–there were “essay questions” so a significant amount of text was necessary. “Preparing for that helped a lot” according to the project leader who submitted the form. The statement of goals from the project proposal was a great starting point for drafting the submission.

Similarly, it was a good idea to draft ahead of time the LinkedIn post unveiling the ChemGraph-IR video.

Both of these were also example of another tip: If you notice something needs to be done, do it yourself if you can, and share the results with the team so they can improve it.

Prototype the output

Even before the code is ready, create a way to test the ultimate output of your project. For example, I created a simple Jupyter Notebook which prompted ChemGraph with “What is the infrared absorption spectrum of water?” This provided multiple benefits:

I ran the prompt and found that the existing code already provided an answer based on other knowledge from the LLM itself. This alerted the team that we might need to tell the LLM not to use its background knowledge, and instead use the new code in progress.
The existing code output the steps used, letting the team know what preparatory steps were taken, which could inform where the new code should pick up.
After my pull request was merged, it gave team members a way to check the end-to-end output of the in-progress code.
It provided a starting point for prompt engineering: A team member later suggested we add “using ASE and TBLite” (the particular tools used to calculate the spectrum) to the end of the prompt.

Commit code early and often

Even if you don’t think your branch isn’t all done, or may not be that useful, submit a pull request when it’s largely complete. I did that and marked it as draft, but the project maintainer wisely merged it. That let others build on it.

And from the consumption side, when I was working on the presentation, it was helpful to have the latest code merged so I could pull it and run the product to produce the output.

Realize that LLM performance can lead to errors

Particular to this type of hackathon, it became apparent that the performance of the LLM can cause “errors.” For example, if the recursion limit was too low, ChemGraph couldn’t generate a molecular structure. Increasing the recursion limit (which increases the cost!) solved the problem. Or you might find that using a less-advanced LLM fails, while another more-advanced LLM works.

Set up credentials early

When you have limited time, it’s important to get participants able to contribute as soon as possible. Setting up credentials such as a API token for an LLM lets people get going on their own.

Keep track of people and their roles

Our team had members sign up for their primary and secondary roles on a shared spreadsheet. That was a great idea because we knew who to talk to about a given aspect of the project.

In fact, it would have been useful to analyze that sheet and notice that no one had signed up to work on the presentation. In the end it worked out because a couple of us took on the presentation at the start of the second day, but ideally someone should have thought about the presentation from the beginning.

Acknowledgements

Thanks to everyone on the ChemGraph-IR team, particularly the organizers Murat Keçeli and Thang Pham! And to Ben Blaiszik for organizing the hackathon, and David Elbert and Mohd Zaki for hosting me at The Johns Hopkins University in Baltimore.

Check out this article in Science about the hackathon.

Murcko Scaffolds Post on the RDKit Blog: An LLM Circle

2025-07-16T00:00:00+00:00

How I came to write a simple blog post and the effects it had.

An LLM Chat Plants the Seed

To come up with small projects to do in preparation for job interviews (contact me if you need a cheminformatics or scientific software developer!), I interactively asked ChatGPT for suggestions. It suggested

Cheminformatics-Flavored Algorithm Problem A classic coding challenge but wrapped in a domain-relevant context.

Example Prompt:

“You are given a list of SMILES strings. Write a function that identifies and counts the number of unique scaffolds using the Bemis-Murcko scaffold definition. Use RDKit.”

The Murcko Scaffolds Blog Post

I wrote code to find the Murcko scaffold for a set of molecules, then determine the count of unique scaffolds. That answered the ChapGPT challenge.

But as usual, I wanted to visualize the results. So I grouped the molecules by scaffold, then used MolsMatrixToGridImage (which I contributed to the RDKit) to display each scaffold as a row with its molecules to the right.

I contributed the Murcko Scaffolds Tutorial post to the RDKit blog. Thanks to Greg Landrum for publishing the post on that blog!

Another LLM Tool is Applied to Reproduce My Visualization

A couple weeks later, I saw a LinkedIn post about an RDKit model context protocol (MCP) server whose molecular visualization looked awfully familiar: A row for each scaffold with its molecules to the right. The authors indeed used my blog post as a test case of their RDKit Copilot!

Put Things Out Into the World, No Matter How Simple

I love that my simple code and visualization was an inspiration for others to test their new tool. It shows that you never know what effect it will have when you put something out into the world: Someone else may take it in a direction you never expected.

It also confirms what I’ve found in other ways: Simple ideas make good blog posts. It’s easy to think that you have to do something very complex to make a contribution: People have asked me if they need to come up with very advanced ideas for blog posts. But as I’ve written in How to Write Cheminformatics Blog Posts, coming up with ideas can be as simple as going through a tutorial and building on it. In fact, actually I’ve found via Google Search Console that my very first blog post, Find and Highlight the Maximum Common Substructure Between a Set of Molecules Using RDKit is the most visited from searches. Presumably this is because many people want to accomplish the simple task of finding a maximum common substructure.

So no matter how simple the idea, if you learn something by coding it, others probably will too, so go ahead and blog about it.

If your inspiration leads you to blog about something involved like tautomers, that’s fine too! Just don’t think you should reject an idea because it’s too simple.

Skeletal Editing with Reaction SMARTS and RDKit Atom Substitution

2025-06-24T00:00:00+00:00

This tutorial shows two ways to accomplish skeletal editing reactions in the RDKit: By using Reaction SMARTS or atom editing. As my collaborator Dr. Phyo Phyo Zin posted in How to Use Skeletal Editing in Drug Discovery: Key Benefits & Applications, “skeletal editing enables chemists to modify the core structure of existing molecules by swapping, deleting, inserting, or rearranging atoms.” For generality, Reaction SMARTS is best because it can match molecular patterns, enabling one Reaction SMARTS to apply to many reactants. For a simple reaction like pyrylium → pyridinium, we can accomplish the same thing using atom editing on an RDKit read-write molecule.

Here’s the skeletal-editing reaction that we’ll be working with, pyrylium → pyridinium:

Here’s my previous tutorial on skeletal editing.

Open this notebook in Google Colab so you can run it without installing anything on your computer

Code foundation

This section is taken from our previous tutorial on skeletal editing. It installs the RDKit and defines functions to run and plot reactions using Reaction SMARTS.

%%capture
%pip install rdkit

from rdkit import Chem
from rdkit.Chem import AllChem, Draw
from rdkit.Chem import rdqueries
from rdkit.Chem.Draw import IPythonConsole

from IPython.display import display

def run_rxn(rxn_smarts, mol):
    rxn = AllChem.ReactionFromSmarts(rxn_smarts)
    print("Reaction:")
    display(rxn)
    try:
        products = rxn.RunReactants((mol,))
    except IndexError:
        raise IndexError(
            "Reaction failed; the starting material may not match the reaction SMARTS"
        )
    return products

def plot_rxn(rxn_smarts, mol):
    try:
        products = run_rxn(rxn_smarts, mol)
    except IndexError as e:
        print(e)
        return
    print("Reactant:")
    display(mol)
    if not products:
        print("No products")
        return
    print("Distinct products:")
    product_smls = set()
    for product in products:
        product_mol = product[0]
        Chem.SanitizeMol(product_mol)
        # Get the canonical SMILES string of the product molecule
        this_sml = Chem.MolToSmiles(product_mol)
        if this_sml in product_smls:
            continue
        product_smls.add(this_sml)
        print(Chem.MolToSmiles(product_mol))
        display(product_mol)
    return product_smls

Now let’s accomplish the skeletal editing reaction pyrylium → pyridinium using each method. The reaction is from Skeletal Editing: Interconversion of Arenes and Heteroarenes (Joynson and Ball, 2023) figure 13A and can be used in the preparation of Katritzky salts.

Reaction SMARTS

Let’s start by defining the SMARTS pattern for the starting material. It’s a six-membered aromatic ring with a hetero atom, specifically oxygen. The oxygen will be replaced with a nitrogen (connected to an R group), so let’s provide an atom number for the oxygen. We’ll visualize the SMARTS pattern as a molecule so we can check that it matches the starting material:

pyrylium_smarts = "[o+r6:1]"
pyrylium_smarts_mol = Chem.MolFromSmarts(pyrylium_smarts)
pyrylium_smarts_mol

To ensure this Reaction SMARTS is limited to reactants matching the published pattern, we specified that

the oxygen is in an aromatic ring using lower-case o
the oxygen has a positive formal charge using +
the oxygen is in a six-membered ring using r6

Now we can use that reactant SMARTS and transform it to the product using RDKit’s Reaction SMARTS format:

[Reactants]»[Products]

pyrylium_to_pyridinium = f"{pyrylium_smarts}>>[n+:1]-[R]"
pyrylium_to_pyridinium_rxn = AllChem.ReactionFromSmarts(pyrylium_to_pyridinium)
pyrylium_to_pyridinium_rxn

Let’s make sure that our starting material (represented with a Kekulé structure)

pyrylium_sml = "C1=[O+]C=CC=C1"
pyrylium = Chem.MolFromSmiles(pyrylium_sml)
pyrylium

matches the SMARTS pattern:

pyrylium.HasSubstructMatch(pyrylium_smarts_mol)

True

Specificity of the reactant SMARTS pattern

Based on our specification that the oxygen is in an aromatic ring, has a positive formal charge, and is in a six-membered ring, let’s check that some molecules similar to pyrylium don’t match the reactant SMARTS.

Benzene shouldn’t match the reactant SMARTS pattern because benzene lacks the oxygen in the aromatic ring:

benzene_sml = "c1ccccc1"
benzene = Chem.MolFromSmiles(benzene_sml)
benzene

benzene.HasSubstructMatch(pyrylium_smarts_mol)

False

The result False confirms that benzene doesn’t match the reactant SMARTS pattern.

Nor should tetrahydropyran because it’s not aromatic and the oxygen atom doesn’t have a positive charge:

tetrahydropyran_sml = "C1OCCCC1"
tetrahydropyran = Chem.MolFromSmiles(tetrahydropyran_sml)
tetrahydropyran

tetrahydropyran.HasSubstructMatch(pyrylium_smarts_mol)

False

We could have a 10-membered ring which is aromatic, but it won’t match the reactant SMARTS pattern because we specified a six-membered ring:

o_arene10_sml = "C1=CC=[O+]C=CC=CC=C1"
o_arene10 = Chem.MolFromSmiles(o_arene10_sml)
o_arene10

o_arene10.HasSubstructMatch(pyrylium_smarts_mol)

False

Whereas if we create a different reactant SMARTS which specifies only that the oxygen be aromatic, and not the ring size, our 10-membered ring would match:

pyrylium_smarts_any_ring_size = "[o+:1]"
pyrylium_smarts_any_ring_size_mol = Chem.MolFromSmarts(pyrylium_smarts_any_ring_size)
o_arene10.HasSubstructMatch(pyrylium_smarts_any_ring_size_mol)

True

This eight-membered ring with alternating double and single bonds should not match the original reactant pattern because it’s anti-aromatic:

o_antiarene8_sml = "C1=[O+]C=CC=CC=C1"
o_antiarene8 = Chem.MolFromSmiles(o_antiarene8_sml)
o_antiarene8

o_antiarene8.HasSubstructMatch(pyrylium_smarts_mol)

False

Run the Reaction SMARTS on pyrylium

Having established the Reaction SMARTS’ specificity, let’s plot the reaction for pyrylium:

product_smls = plot_rxn(pyrylium_to_pyridinium, pyrylium)

Reaction:

Reactant:

Distinct products:
*[n+]1ccccc1

To later compare to the atom-editing result, let’s save the product SMILES. There’s only one SMILES in the set, so we can pop it off.

product_smls

{'*[n+]1ccccc1'}

product_from_rxn_smarts = product_smls.pop()
product_from_rxn_smarts

'*[n+]1ccccc1'

Alternatively, we can use the RDKit’s ChemicalReaction() to visualize the reaction in a one-row format. We first add the reactant and product templates to the specific reaction, then we can visualize the reaction:

product_mol = Chem.MolFromSmiles(product_from_rxn_smarts)
specific_rxn = AllChem.ChemicalReaction()
specific_rxn.AddReactantTemplate(pyrylium)
specific_rxn.AddProductTemplate(product_mol)

img = Draw.ReactionToImage(specific_rxn, subImgSize=(300, 300))
display(img)

Generality of the Reaction SMARTS

We can verify that our Reaction SMARTS is written in a general manner by successfully running the reaction on a substituted pyrylium. Here we use the 1* label simply as a marker; the identity of the ortho-substituted moiety might or might not affect whether this skeletal editing reaction would work in practice.

pyrylium_substituted_sml = "[1*]C1=[O+]C=CC=C1"
pyrylium_substituted = Chem.MolFromSmiles(pyrylium_substituted_sml)
pyrylium_substituted

plot_rxn(pyrylium_to_pyridinium, pyrylium_substituted)

Reaction:

Reactant:

Distinct products:
*[n+]1ccccc1[1*]

{'*[n+]1ccccc1[1*]'}

The ortho substitution on the arene ring is preserved, so the Reaction SMARTS worked as desired.

RDKit atom substitution

Because this is a fairly simple atom-for-atom replacement (though with the addition of an R group on the new, nitrogen atom), we can accomplish the same thing by modifying the molecule. We start by creating a read-write molecule from the original (read-only) molecule and verifying that it’s the same as the original. We’ll display atom indices because we’ll use them to address atoms in the following code.

IPythonConsole.drawOptions.addAtomIndices = True

pyrylium_rwm = Chem.RWMol(pyrylium)
pyrylium_rwm

To programmatically identify the oxygen atom by its atomic number of 8, we create and run a query. There’s only one oxygen atom, so we can simply get the first atom matching the query.

hetero_atom_query = rdqueries.AtomNumEqualsQueryAtom(8)
hetero_atom = pyrylium_rwm.GetAtomsMatchingQuery(hetero_atom_query)[0]
hetero_atom_idx = hetero_atom.GetIdx()
hetero_atom_idx

Now we’ll replace the oxygen atom with a nitrogen:

pyrylium_rwm.ReplaceAtom(index=hetero_atom_idx, newAtom=Chem.Atom(7))
Chem.SanitizeMol(pyrylium_rwm)
pyrylium_rwm

To add the R group to the nitrogen, we use the read-write molecule’s AddAtom method. We start by just adding the atom * representing the R group. If we leave it at that, the atom is disonnected from the rest of the molecule (note the * floating above the rest of the molecule):

added_atom_idx = pyrylium_rwm.AddAtom(Chem.Atom("*"))
pyrylium_rwm

So we need to add a bond between the nitrogen and the R group.

pyrylium_rwm.AddBond(
    beginAtomIdx=hetero_atom_idx,
    endAtomIdx=added_atom_idx,
    order=Chem.BondType.SINGLE,
)
pyrylium_rwm

Now let’s clean up the molecule by assigning the nitrogen to have no hydrogen atoms, and its charge to +1. First let’s programmatically find the nitrogen atom. Note that its index is the same as the oxygen atom had, but we’ll find the nitrogen atom (as an object) so we can adjust its properties later.

n_query = rdqueries.AtomNumEqualsQueryAtom(7)
n_at = pyrylium_rwm.GetAtomsMatchingQuery(n_query)[0]
n_at.SetNoImplicit(True)
n_at.SetNumExplicitHs(0)
n_at.SetFormalCharge(1)

Next we’ll draw the molecule, highlighting the substituted heteroatom, the new R group, and the bond between them.

new_bond_idx = pyrylium_rwm.GetBondBetweenAtoms(
    hetero_atom_idx, added_atom_idx
).GetIdx()
Draw.MolToImage(
    mol=pyrylium_rwm,
    highlightAtoms=[hetero_atom_idx, added_atom_idx],
    highlightBonds=[new_bond_idx],
)

Finally, we’ll convert the read-write molecule back to a read-only molecule, sanitize it, and display it.

product_from_atom_subs = pyrylium_rwm.GetMol()
Chem.SanitizeMol(product_from_atom_subs)
product_from_atom_subs

Equivalency of the two methods

Let’s verify that the product is the same as that from the Reaction SMARTS.

product_from_rxn_smarts == Chem.MolToSmiles(product_from_atom_subs)

True

Retrosynthetically Decompose a Molecule In a Tree Structure Using Recap in RDKit–Update to Annotate Diagram Programmatically

2025-05-04T00:00:00+00:00

This is an update of a Recap retrosynthetic tree blog post to programmatically annotate the diagram. Previously, I exported the drawing from the RDKit and manually annotated the it with another program. After reading Greg Landrum’s RKDit blog post Drawing on drawings, I wanted to automate annotating the tree.

Retrosynthetic analysis involves decomposing a target molecule into a set of fragments that could be combined to make the parent molecule using common reactions. The Recap algorithm by X. Lewell, D. Judd, S. Watson, and M. Hann accomplishes that. Recap is implemented in the RDKit cheminformatics Python package.

Open this notebook in Google Colab so you can run it without installing anything on your computer

RDKit helpfully provides a RecapHierarchyNode structure of nodes, where the keys are SMILES strings corresponding to the fragment, and the values are nodes containing child fragments. However, it is not easy to visualize the results, because the results are SMILES strings and the hierarchy is not shown visually.

This utility function molecule_recap_tree allows you to visualize both the fragments and their hierarchy. Here is an example, of [6-(Hydroxymethoxy)pyridin-2-yl]oxymethanol, annotated to explain the hierarchy:

As compared to the same molecule using the RecapDecompose function directly, showing only the immediate children:

from IPython.display import SVG
from rdkit import Chem
from rdkit.Chem import Draw, Recap
from rdkit import Geometry

parent = "n1c(OCO)cccc1-OCO"
molecule = Chem.MolFromSmiles(parent)
hierarch = Recap.RecapDecompose(molecule)
ks = hierarch.children.keys()
sorted(ks)
molecule

def concat_grids_horizontally(
    grid1: list[list[str]], grid2: list[list[str]]
) -> list[list[str]]:
    """Concatenate two nested lists horizontally, for example
    inputs [['a'],['b'],['c']] and [['d'], ['e'], ['f']]
    produce [['a', 'd'], ['b', 'e'], ['c', 'f']]

    :returns: The combined grid, a two-deep nested list of strings
    :param grid1: The first grid, a two-deep nested list of strings
    :param grid2: The second grid, a two-deep nested list of strings
    """
    if grid1 == [[]]:
        combined = grid2
    elif grid2 == [[]]:
        combined = grid1
    else:
        combined = []
        for row_counter in range(len(grid1)):
            combined += [grid1[row_counter] + grid2[row_counter]]
    return combined


class NonBinTree:
    """
    Nonbinary tree class
    Note that this class is not designed to sort nodes as they are added to the tree;
    the assumption is that they should be ordered in the order added
    Adapted from https://stackoverflow.com/questions/60579330/non-binary-tree-data-structure-in-python#60579464
    """

    def __init__(self, val: str):
        """Create a NonBinTree instance"""
        self.val = val
        self.nodes = []

    def add_node(self, val: str):
        """Add a node to the tree and return the new node"""
        self.nodes.append(NonBinTree(val))
        return self.nodes[-1]

    def __repr__(self) -> str:
        """Print out the tree as a nested list"""
        return f"NonBinTree({self.val}): {self.nodes}"

    def get_ncols(self) -> int:
        """Get the number of columns in the tree"""
        self.ncols = 0
        if len(self.nodes) > 0:
            # If there are nodes under this one, call get_ncols on them recursively
            for node in self.nodes:
                self.ncols += node.get_ncols()
        else:
            # If there are no nodes under this one, add 1 for this node
            self.ncols += 1
        return self.ncols

    def get_max_depth(self) -> int:
        """Get the maximum depth of the tree"""
        max_depth = 0
        if len(self.nodes) > 0:
            for node in self.nodes:
                this_depth = node.get_max_depth()
                max_depth = max(this_depth + 1, max_depth)
        else:
            max_depth = max(1, max_depth)
        self.max_depth = max_depth
        return self.max_depth

    def get_grid(self) -> list[list[str]]:
        """
        Get a two-dimensional grid where
        each row is a level in the fragment hierarchy, and
        the columns serve to arrange the fragments horizontally
        """
        # Call methods to calculate self.ncols and self.max_depth
        self.get_ncols()
        self.get_max_depth()

        # Create top row: Node value, then the rest of columns are blank (empty strings)
        grid = [[self.val] + [""] * (self.ncols - 1)]

        n_nodes = len(self.nodes)

        if n_nodes > 0:
            nodes_grid = [[]]

            # Iterate through the child nodes
            for node_counter, node in enumerate(self.nodes):
                # Recursively call this function to get the grid for children
                node_grid = node.get_grid()

                # Add spacer rows if needed
                node_grid_rows = len(node_grid)
                rows_padding = self.max_depth - node_grid_rows - 1
                for padding in range(rows_padding):
                    node_grid += [[""] * len(node_grid[0])]

                nodes_grid = concat_grids_horizontally(nodes_grid, node_grid)

            grid += nodes_grid

        return grid

def get_children(
    base_node: Chem.Recap.RecapHierarchyNode, root: NonBinTree = None
) -> NonBinTree:
    """
    Convert an RDKit RecapHierarchyNode into a NonBinTree by
    traversing the RecapHierarchyNode, getting all its children recursively, and adding them to a NonBinTree

    :returns: NoBinTree containing the Recap hierarchy
    :param base_node: The RDKit RecapHierarchyNode
    :param root: The NoBinTree containing only the root node
    """
    for smiles, node in base_node.children.items():
        added_tree_node = root.add_node(smiles)
        children = node.children.keys()
        # Sort the children nodes to get consistent ordering
        children = sorted(children)
        if len(children) > 0:
            get_children(node, added_tree_node)
    return root

Here comes the new part, annotating the tree. It has two parts:

The Increasing fragmentation text and arrow on the left edge from the top to the bottom of the tree. This is simple to draw because it applies to the tree as a whole, so we just need to calculate where to draw the text and arrow once.
Draw an arrow from each parent to child (fragment). This requires more logic to detect each parent-child pair.
1. RDKit’s DrawArrow specifies the arrowhead size as a fraction of the line’s length. I want all the arrowheads to be the same size, so I set the fraction to be inversely proportional to the line length using the arrow_length logic.
2. I decided to drop the Fragments labels under the arrows because the Increasing fragmentation label explains what the vertical axis means.

To draw on the drawing, we use DrawMolecules rather than MolsMatrixToGridImage or MolsToGridImage as in the original blog post.

def annotate_recap_tree(
    molsMatrix: list[list[Chem.Mol]],
    fragment_grid: list[list[str]],
    recap_plot: list[str],
):
    """
    Annotate the Recap tree with arrows and labels

    :returns: RDKit grid image
    :rtype: RDKit grid image
    :param molsMatrix: The matrix of RDKit molecules
    :param fragment_grid: The matrix of fragment SMILES strings
    :param recap_plot: The list of Recap SMILES strings
    """
    nRows = len(molsMatrix)
    nCols = len(molsMatrix[-1])
    panelx = 250
    panely = 250
    width = panelx * nCols
    height = panely * nRows
    d2d = Draw.MolDraw2DSVG(width, height, panelx, panely)
    dopts = d2d.drawOptions()
    dopts.legendFontSize = 20

    recap_plot_mols = [Chem.MolFromSmiles(sml) for sml in recap_plot]

    # Draw the molecule and fragments
    d2d.DrawMolecules(
        recap_plot_mols,
        legends=recap_plot,
    )

    d2d.SetLineWidth(4)
    font_size = 20
    d2d.SetFontSize(font_size)

    # Use gray arrows
    arrowColor = (0.4, 0.4, 0.4)

    # Arrow for "Increasing Fragmentation"
    p1 = Geometry.Point2D(panelx * 3 / 4, panely / 4)
    p2 = Geometry.Point2D(panelx * 3 / 4, height - 10)
    d2d.DrawArrow(p1, p2, color=arrowColor, rawCoords=True, asPolygon=False, frac=0.01)

    # Text "Increasing Fragmentation"
    midp = Geometry.Point2D()
    midp.x = panelx * 1 / 4
    midp.y = height / 2
    d2d.DrawString("Increasing", midp, rawCoords=True)
    midp.y += font_size
    d2d.DrawString("fragmentation", midp, rawCoords=True)

    # Constant used to make arrowhead size constant
    # regardless of the length of the arrow
    arrow_length = 10

    # Iterate through fragment grid to draw the Fragments and their arrows
    for row_index, row in enumerate(fragment_grid[:-1]):
        # Find the non-empty entries in this row
        this_row_filled_indexes = set()
        for column_index, entry in enumerate(row):
            if entry:
                this_row_filled_indexes.add(column_index)
        # Add a ghost entry for beyond the last column
        this_row_filled_indexes.add(len(row))

        # Iterate through the columns in this row
        #   and draw an arrow for each fragment that has children
        for col_index, sml in enumerate(row):
            if not col_index:
                # Skip the first column, which is the whole-chart label "Increasing Fragmentation"
                continue

            # The next sibling index is the index of the next fragment in the same row
            #   that is not blank (empty string)
            next_sibling_index = min(
                index for index in this_row_filled_indexes if index > col_index
            )

            # If no molecule at this row_index, col_index, skip drawing an arrow
            if not sml:
                continue

            # If no molecule directly below this row_index, col_index, skip drawing an arrow
            if not fragment_grid[row_index + 1][col_index]:
                continue

            # Set the starting point of all arrows for this parent
            #   to be below the center of the current fragment
            p1 = Geometry.Point2D(
                panelx * (col_index + 1 / 2), row_index * panely + panely * 1
            )
            for child_col_index in range(col_index, next_sibling_index):
                # If no molecule directly below this row_index, child_col_index, skip drawing an arrow
                if not fragment_grid[row_index + 1][child_col_index]:
                    continue
                # Set the end point of this arrow to be above the center of the child (fragment)
                p2 = Geometry.Point2D(
                    p1.x + panelx * (child_col_index - col_index),
                    row_index * panely + panely * 1.25,
                )

                # Set the arrowhead size to be constant
                # regardless of the length of the arrow
                length = ((p2.x - p1.x) ** 2 + (p2.y - p1.y) ** 2) ** 0.5
                try:
                    frac = arrow_length / length
                except ZeroDivisionError:
                    frac = 0

                d2d.DrawArrow(
                    p1, p2, color=arrowColor, rawCoords=True, asPolygon=False, frac=frac
                )
    d2d.FinishDrawing()
    svg = d2d.GetDrawingText()
    return svg

def molecule_recap_tree(smiles: str, name: str = "", verbose=False) -> SVG:
    """
    Draw a molecular fragmentation tree using the Recap algorithm given in
    https://www.semanticscholar.org/paper/RECAP-%E2%80%94-Retrosynthetic-Combinatorial-Analysis-A-New-Lewell-Judd/fbfb10d1f63aa803f6d47df6587aa0e41109f5ee

    :returns: RDKit grid image, and (if verbose=True) RDKit molecule of parent molecule, Recap hierarchy, nonbinary tree hierarchy, and fragment grid
    :rtype: RDKit grid image, and (if verbose=True) rdkit.Chem.rdchem.Mol, rdkit.Chem.Recap.RecapHierarchyNode, NonBinTree, list[list[str]]
    :param smiles: The SMILES string of the molecule to be fragmented
    :param name: The name of the parent molecule; if not supplied, it will be labled with its SMILES string
    :param verbose: Whether to return verbose output; default is False so calling this function will present a grid image automatically
    """
    molecule = Chem.MolFromSmiles(smiles)
    RecapHierarchy = Recap.RecapDecompose(molecule)
    root = NonBinTree(RecapHierarchy.smiles)
    molecule_nonbinary_tree = get_children(RecapHierarchy, root)
    fragment_grid = molecule_nonbinary_tree.get_grid()
    # Add a blank element at the beginning of each row to make room for the left legend
    for row in fragment_grid:
        row.insert(0, "")
    recap_plot = [item for sublist in fragment_grid for item in sublist]
    recap_labels = [item for sublist in fragment_grid for item in sublist]
    if name:
        recap_labels[0] = name

    molsMatrix = [
        [Chem.MolFromSmiles(smile) for smile in sublist] for sublist in fragment_grid
    ]

    drawing = annotate_recap_tree(molsMatrix, fragment_grid, recap_plot)

    if verbose:
        return (
            drawing,
            molecule,
            RecapHierarchy,
            molecule_nonbinary_tree,
            fragment_grid,
            recap_plot,
            molsMatrix,
        )
    else:
        return drawing

Here is the output annotated with RDKit:

drawing = molecule_recap_tree(parent, "Parent molecule")
SVG(drawing)

Stars (“dummy atoms”) show points where the molecule was fragmented.

The key RDKit commands that molecule_recap_tree uses are:

RecapDecompose to decompose the parent molecule into successive fragments
MolsToGridImage to draw the fragment hierarchy in a grid, and label the fragments with their SMILES strings

Get Additional Data

If you want molecule_recap_tree to return not just the grid image, but also the parent molecule, the RDKit Recap hiearchy node, the non-binary tree hierarchy, and grid of fragment hierarchy, set verbose=True:

(
    drawing,
    molecule,
    RecapHierarchy,
    molecule_nonbinary_tree,
    fragment_grid,
    recap_plot,
    molsMatrix,
) = molecule_recap_tree(parent, "Parent molecule", verbose=True)

You then must explicitly call the hierarchy image to draw it:

SVG(drawing)

molecule is the RDKit parent molecule:

molecule

RecapHierarchy is the RDKit Recap hierarchy, which returns the top-level node, which you could traverse to obtain the hierarchy:

RecapHierarchy

molecule_nonbinary_tree is the NonBinTree hierarchy, which contains the same hierarchy as the RecapHierarchy, and gives all the nodes directly:

molecule_nonbinary_tree

NonBinTree(OCOc1cccc(OCO)n1): [NonBinTree(*CO): [], NonBinTree(*c1cccc(OCO)n1): [NonBinTree(*CO): [], NonBinTree(*c1cccc(*)n1): []]]

fragment_grid is the two-dimensional fragment grid, where each row is a level in the fragment hierarchy, and the columns serve to arrange the fragments horizontally:

fragment_grid

[['', 'OCOc1cccc(OCO)n1', '', ''],
 ['', '*CO', '*c1cccc(OCO)n1', ''],
 ['', '', '*CO', '*c1cccc(*)n1']]

Or, more nicely formatted into columns and with empty strings not shown:

# Get the longest string (cell) in the grid
max_length = max(
    len(item) for sublist in fragment_grid for item in sublist if item != ""
)

for row in fragment_grid:
    f_str = ""
    for col in row[1:]:
        # Make each string the same length, namely the max length
        if col == "":
            f_str += f"{col:<{max_length}} "
        else:
            # Center the string in a field of max_length
            f_str += f"{col:<{max_length}} "
    print(f"{f_str}")

OCOc1cccc(OCO)n1                                   
*CO              *c1cccc(OCO)n1                    
                 *CO              *c1cccc(*)n1     

Here’s an example of a larger molecule with more fragments.

parent = "n1c(OCCCO)cccc1OCO"
drawing = molecule_recap_tree(parent, "Parent molecule")
SVG(drawing)

Note About Hierarchies

Another approach I could have taken was to extend the class rdkit.Chem.Recap.RecapHierarchyNode, rather than use the NonBinaryTree class. Using the NonBinaryTree class allows for different kinds of hierarchies to be depicted as a grid, as I demonstrate in other posts Visualizing Nonbinary Trees: Classification of Chemical Isomers and Draw a Mass Spectrometry Fragmentation Tree Using RDKit.

Tutorial for Skeletal Editing in Drug Discovery

2025-04-22T00:00:00+00:00

This notebook is the result of a collaboration between myself, Dr. Jeremy E. Monat, and Dr. Phyo Phyo Zin. This is a hands-on tutorial that guides readers through implementing skeletal editing transformations using Python and RDKit, and Phyo Phyo focuses on the high-level principles and applications in the blog post How to Use Skeletal Editing in Drug Discovery: Key Benefits & Applications. Together, our goal is to provide both a conceptual understanding and practical approach to this emerging field.

Open this notebook in Google Colab so you can run it without installing anything on your computer

Code foundation

First we install the RDKit.

%%capture
!pip install rdkit

Next we import the necessary modules.

from rdkit import Chem
from rdkit.Chem import AllChem
from rdkit.Chem.Draw import IPythonConsole

from IPython.display import display

Now we set up drawing options for the blog post.

IPythonConsole.molSize = 400, 200

Let’s define a function to display and run a reaction with RunReactants using the RDKit’s Reaction SMARTS format.

def run_rxn(rxn_sml, mol):
    rxn = AllChem.ReactionFromSmarts(rxn_sml)
    print("Reaction:")
    display(rxn)
    try:
        products = rxn.RunReactants((mol,))
    except IndexError:
        raise IndexError(
            "Reaction failed; the starting material may not match the reaction SMARTS"
        )
    return products

Now let’s fold that into a larger function that also displays the reactant and distinct products. Because of the way SMARTS matches parts of a molecule, we might have multiple identical products, so we only want to show each product once.

def plot_rxn(rxn_sml, mol):
    try:
        products = run_rxn(rxn_sml, mol)
    except IndexError as e:
        print(e)
        return
    print("Reactant:")
    display(mol)
    if not products:
        print("No products")
        return
    print("Distinct products:")
    product_smls = set()
    for product in products:
        product_mol = product[0]
        Chem.SanitizeMol(product_mol)

        # Get the canonical SMILES string of the product molecule
        this_sml = Chem.MolToSmiles(product_mol)

        # If this product has already been made, don't display it again
        if this_sml in product_smls:
            continue
        product_smls.add(this_sml)
        print(Chem.MolToSmiles(product_mol))
        display(product_mol)
    return product_smls

Skeletal Editing Reactions Used in Drug Discovery Blog Post

Here are three skeletal editing reactions used in Phyo Phyo’s blog post. In each case, we give

the general reaction and apply it to a generic reactant, then
a modified reaction and apply it to a molecule related to drug discovery.

Converting quinolines to quinazolines

This reaction is from Woo et al., 2023, scheme 1d.

quinoline = "*c1nc2c(cccc2)cc1"
quinoline_mol = Chem.MolFromSmiles(quinoline)
quinoline_mol

quinoline_to_quinazoline = "*-c1nc2c(cccc2)c[c:1]1>>*-c1nc2c(cccc2)c[n:1]1"
product_smls = plot_rxn(quinoline_to_quinazoline, quinoline_mol)

Reaction:

Reactant:

Distinct products:
*c1ncc2ccccc2n1

Skeletal Editing for Talnetant

quinoline_to_quinazoline_modified = "[cX3:1]1[n:2][c:3]2[c:4]([c:5][c:6][c:7][c:8]2)[c:9][c:10]1>>[cX3:1]1[n:2][c:3]2[c:4]([c:5][c:6][c:7][c:8]2)[c:9][n:10]1"
talnetant_smi = "c1ccccc1c2cc(C(=O)NC(CC)c4ccccc4)c3ccccc3n2"
talnetant_mol = Chem.MolFromSmiles(talnetant_smi)
product_smls = plot_rxn(quinoline_to_quinazoline_modified, talnetant_mol)

Reaction:

Reactant:

Distinct products:
CCC(NC(=O)c1nc(-c2ccccc2)nc2ccccc12)c1ccccc1

Converting pyrrolidines to cyclobutanes

This reaction is from Hui et al., 2021, Scheme 1.

pyrrolidine = "*C(=O)C1CCCN1"
pyrrolidine_mol = Chem.MolFromSmiles(pyrrolidine)
pyrrolidine_mol

pyrrolidine_to_cyclobutane = "*-C(=O)-[C:1]1-C-C-[C:2]-N-1>>*-C(=O)-[C:1]1-C-C-[C:2]-1"
product_smls = plot_rxn(pyrrolidine_to_cyclobutane, pyrrolidine_mol)

Reaction:

Reactant:

Distinct products:
*C(=O)C1CCC1

pyrrolidine_to_cyclobutane_modified = (
    "[C:5](=O)-[C:1]1-[C:3]-[C:4]-[C:2]-N-1>>[C:5](=O)-[C:1]1-[C:3]-[C:4]-[C:2]-1"
)
starting_smi = "c1cc(Br)ccc1C2NC(C(=O)OC)C(c3ccccc3)C(C(=O)OC)2"
starting_mol = Chem.MolFromSmiles(starting_smi)
product_smls = plot_rxn(pyrrolidine_to_cyclobutane_modified, starting_mol)

Reaction:

Reactant:

Distinct products:
COC(=O)C1C(c2ccccc2)C(C(=O)OC)C1c1ccc(Br)cc1

Converting nitroarenes into azepanes

This reaction is from Mykura et al., 2024.

nitroarene = "O=[N+]([O-])c1ccccc1"
nitroarene_mol = Chem.MolFromSmiles(nitroarene)
nitroarene_mol

nitroarene_to_azepane = "O=[N+](-[O-])-[c:2]1[c:3]cccc1>>[C:2]-1-[N]-[C:3]-C-C-C-C1"
product_smls = plot_rxn(nitroarene_to_azepane, nitroarene_mol)

Reaction:

Reactant:

Distinct products:
C1CCCNCC1

nitroarene_to_azepane_modified = "O=[N+](-[O-])-[c:2]1[c:3][c:4][c:5][c:6][c:7]1>>[C:2]-1-[N]-[C:3]-[C:4]-[C:5]-[C:6]-[C:7]1"
starting_smi = "CCC(=O)N(c1ccccc1)c3cccc([N+](=O)[O-])c3"
starting_mol = Chem.MolFromSmiles(starting_smi)
product_smls = plot_rxn(nitroarene_to_azepane_modified, starting_mol)

Reaction:

Reactant:

Distinct products:
CCC(=O)N(c1ccccc1)C1CCCNCC1

CCC(=O)N(c1ccccc1)C1CCCCNC1

Phyo Phyo discusses the distinction between these two products and the factors favoring one over the other in the blog post How to Use Skeletal Editing in Drug Discovery: Key Benefits & Applications.

Update

Here’s a follow-up tutorial: Skeletal Editing with Reaction SMARTS and RDKit Atom Substitution

Drawing molecules with EPAM’s Indigo cheminformatics library

2025-01-11T00:00:00+00:00

Exploring other cheminformatics toolkits besides the RDKit, I wanted to try EPAM Indigo Toolkit. The Indigo Toolkit is free and open-source with Apache License 2.0, so it can be used in proprietary software. I was unable to find simple examples of drawing molecules in a Python Jupyter Notebook, so here’s how to do that. This post also demonstrates how to save molecular images to a file.

In general the official documentation for Indigo is much less than for the RDKit, as are the examples posted by users online. As a form of documentation, below I specify Indigo argument names, and am more pedantic about including code blocks and outputs.

Open this notebook in Google Colab so you can run it without installing anything on your computer

Import and initialize Indigo modules

Indigo works a little differently than the RDKit: With Indigo, you start by creating an instance of the class for Indigo or its renderer; then you can use the class’s methods.

from indigo import Indigo
from indigo.renderer import IndigoRenderer

# Initialize Indigo and IndigoRenderer
indigo = Indigo()
renderer = IndigoRenderer(indigo)

To display the images in a Jupyter Notebook, we use IPython display modules.

from IPython.display import display, SVG, Image

Set an Indigo option for all images

As a first example of performing an operation in Indigo, let’s set a drawing option that will apply to the rest of this post. This option, render-margins, sets the “Horizontal and vertical margins around the image, in pixels.”

indigo.setOption("render-margins", 10, 10)

It’s helpful to have such fine control of drawing. Compared to the RDKit, I found Indigo’s syntactical structure less transparent because you supply the option name in quotes, meaning you don’t get a list of parameter name options, or autocomplete in an IDE (integrated development environment).

Draw a single molecule with Indigo Renderer

Set up a single molecule

One nice feature of Indigo is that it will convert from a chemical name to a molecule, so you don’t need to compose the SMILES.

name = "3-ethyl-octane"
mol = indigo.nameToStructure(name)

Once you have the molecule, you can get the SMILES using the molecule’s smiles method.

mol.smiles()

'CCC(CCCCC)CC'

Display an Indigo molecular structure in the notebook

I was able to render a molecular structure in a Jupyter Notebook using either PNG or SVG format. In each code block, we start by setting the render-output-format option to the desired graphics format. The Jupyter Notebook command to display the image varies depending on the image format.

indigo.setOption("render-output-format", "png")
img = renderer.renderToBuffer(
    obj=mol,
)
display(Image(img))

indigo.setOption("render-output-format", "svg")
mol_svg = renderer.renderToBuffer(
    obj=mol,
)
display(SVG(mol_svg))

Taking a different approach than the RDKit, Indigo renders terminal carbon atoms explicitly (CH₃ here) rather than implicitly as vertices.

Write an Indigo molecular structure to a file

Similarly, we can save a molecular structure to a file in either PNG or SVG format.

indigo.setOption("render-output-format", "png")
mol.layout()
renderer.renderToFile(
    obj=mol,
    filename="/images/indigo/mol.png",
)

Here’s the image in the PNG file:

indigo.setOption("render-output-format", "svg")
mol.layout()
renderer.renderToFile(
    obj=mol,
    filename="/images/indigo/mol.svg",
)

And the SVG file has the same molecular structure:

Draw a grid of molecules with Indigo Renderer

Often we want to draw a grid of molecular structures to show similarities amongst them, depict the results of a query, etc.

Set up an array of Indigo molecules

First let’s create a list of molecules based on cyclohexane.

names = [
    "cyclohexane",
    "1-methyl-cyclohexane",
    "1,2-dimethyl-cyclohexane",
    "1,3-dimethyl-cyclohexane",
    "1,4-dimethyl-cyclohexane",
    "1,2,3-trimethyl-cyclohexane",
]

In Indigo, we use createArray() to create an array of molecules to later draw in a grid. Let’s set the property grid-comment of each molecule to its name so we can later display its name in the drawing grid.

array = indigo.createArray()
for n in names:
    this_mol = indigo.nameToStructure(n)
    this_mol.layout()
    this_mol.setProperty("grid-comment", n)
    array.arrayAdd(this_mol)

In the options below, title refers to the label under each molecule in the grid. The title will be the molecule’s name because that’s what we previously set grid-comment to. Again, the granular ability to set specific layout properties, such as vertical and horizontal spacing, is nice.

# Set the label under each molecule
indigo.setOption("render-grid-title-property", "grid-comment")

# Set the vertical offset of the label under each molecule
indigo.setOption("render-grid-title-offset", "15")

# Set the spacing between grid cells as horizontal and vertical pixels
indigo.setOption("render-grid-margins", "70, 70")

Display a grid of Indigo molecular structures in the notebook as a PNG

Now we can draw a grid of molecules using renderGridToBuffer. We pass the array of molecules as the objects parameter. Similar to the RDKit, we can specify the number of columns, here with ncolumns. The parameter refatoms can be used to align structures to have the same orientation, though the code below doesn’t set it to anything.

indigo.setOption("render-output-format", "png")
img_grid_png = renderer.renderGridToBuffer(
    objects=array,
    refatoms=None,
    ncolumns=3,
)
display(Image(img_grid_png))

Display a grid of Indigo molecular structures in the notebook as an SVG

The output from this has been intermittently problematic: the molecular structures have come out distorted, perhaps due to some issue with writing or displaying the SVG format. The SVG format specifies graphical elements and their locations, which has the advantages of vector graphics but this possible issue. On the other hand, PNG is a raster-graphics format where the image is specified as a grid of pixels, so it’s not surprising that it doesn’t have the same issue.

indigo.setOption("render-output-format", "svg")
img_grid = renderer.renderGridToBuffer(
    objects=array,
    refatoms=None,
    ncolumns=3,
)
display(SVG(img_grid))

Write a grid of Indigo molecules to an image file

Both PNG and SVG formats are possible.

indigo.setOption("render-output-format", "png")
renderer.renderGridToFile(
    objects=array,
    refatoms=None,
    ncolumns=3,
    filename="/images/indigo/structures.png",
)

Here’s the image in the PNG file:

indigo.setOption("render-output-format", "svg")
renderer.renderGridToFile(
    objects=array,
    refatoms=None,
    ncolumns=3,
    filename="/images/indigo/structures.svg",
)

Here’s the image in the SVG file:

Another resource

You can find other examples of how to use Indigo at Chemistry Toolkit Rosetta Wiki, which has code examples for common operations in cheminformatics toolkits such as Indigo, RDKit, CDK, OpenBabel, OpenEye, CACTVS, Cinfony, and Chemkit. The page Depict a compound as an image is relevant to the goal of this post.

How to Write Cheminformatics Blog Posts

2024-11-21T00:00:00+00:00

As the YouTubers would say, “A lot of you have been asking me about how to write cheminformatics blog posts.” Well, not a lot, but at least a couple! I realized that there’s a pattern to how I write cheminformatics blog posts (16 so far), so I’m sharing that here.

My blog posts are intended to be tutorials that explore a topic, usually with existing tools. I figure out how to accomplish a cheminformatics objective, then share that.

This post is about the creative process more than the technical details and is pitched toward people with a chemistry background.

Throughout this post, I’ll use as an example my last blog post, Why some organic molecules have a color: Correlating optical absorption wavelength with conjugated bond chain length.

Find a jumping-off point

A jumping-off point can be any of these approaches you find interesting:

A tutorial. Examples: Getting Started with the RDKit in Python, RDKit Cookbook, or a cheminformatics blog post by, for example, Takayuki Serizawa.
A talk or presentation. Examples: RDKit User Group Meeting (UGM) recordings.
A dataset. Examples: Experimental database of optical properties of organic compounds, datasets on MoleculeNet.

Example: My background is in optical spectroscopy, so I searched for an optical spectroscopy dataset, and found one with thousands of chromophores and their absorption and emission wavelength maxima. Thanks to relatively new publications such as Nature Scientific Data, there are a good number of datasets available.

Ask yourself “What would I as a chemist want to do with this?”

This is where your chemistry background comes into play. Given the jumping-off point, think about what you as a chemist would want to do with the function or topic.

Your interests will determine what you find interesting as a jumping-off point and what you want to do with it. It’s that combination that means that each cheminformatician will come up with different topics.

Find something compelling to you and follow your curiosity. When you find something you want to work on and can’t put down, you’ve found a good topic.

Example: When I considered the optical dataset, I remembered that there’s a correlation between conjugated bond chain length and absorption wavelength. So I thought about verifying that using this dataset.

Formulate an approach

What chemical information do you want?

I think it’s best to start with the question of what chemical information you want, then figure out a way to calculate it. This order ensures that the topic is chemically relevant.

Example: I needed to determine the longest conjugated bond chain length in a molecule. (Absorption wavelength was already provided, though I wanted to convert it to energy, so that required a calculation.)

What code will you need to write to extract the chemical information?

This is where the cheminformatics part comes in: Translating the chemical information question into a plan for how to get it using code. Sometimes the core function you need is already available, for example in a cheminformatics toolkit, so it’s a matter of applying it to your situation. Other times, you need to write the function yourself, for example by considering a molecule as a graph where the nodes are atoms and the edges are bonds.

Example: The optical dataset provided SMILES, which the RDKit can turn into a molecular graph, so the task was to devise an algorithm that would traverse the conjugated bonds in the molecular graph and determine which were connected to a starting bond. Then find all such chains and keep the longest one in the molecule.

Consult a journal article

It’s often helpful to consult a journal article to get molecular structures, experimental details, etc. I’ve often used my American Chemical Society membership benefit ACS Member Universal Access (50 article accesses per term) to access journal articles, though many recent publications may be available on preprint archive servers now.

Example: I found the optical properties dataset via the journal article that presented it. I also read the journal article that studied the oligomers in the homologous series. This helped me correct the structures in the database and understand the authors’ explanations of trends within that series.

Code your way through the project

Here’s where I start writing code. It’s typically an iterative process where I solve one problem and get results, only to realize I need to solve an additional problem.

Example 1: In finding the longest conjugated chain, I started with one algorithm, but realized it didn’t produce the results that made chemical sense because it wouldn’t include the shared bond between the two rings in naphthalene. So I generalized my conception of a “chain” to include all conjugated bonds adjacent to an existing conjugated bond in the “chain,” which allowed for multiple paths.

Example 2: After that code was working, the results showed that there wasn’t much of a correlation of absorption wavelength against longest conjugated bond chain length if I included thousands of chromophores. So I realized I needed to narrow down to a series of molecules with similar structures by finding chromophores in the dataset with moieties in common.

Make the critical code more efficient

Often, there’s a single cheminformatics operation which takes a significant amount of time on a sizeable dataset. This slow step is worth speeding up, especially because I often run it dozens of times while drafting the blog post.

Example: In the optical properties post, the slow step was traversing each molecular graph to determine its longest conjugated bond chain. I sped this up by only traversing each bond once, rather than using each bond as a starting point and then traversing all connected bonds. I also sped up the operation at the dataframe level: Because it contained multiple rows for many chromophores, I cached the result for each chromophore so it wouldn’t need to be recalculated.

Note that speeding up the code is not the only goal: Because the blog post is intended to be educational, there’s a balance between code speed and clarity. For example, in many posts, I use Polars in eager mode rather than lazy mode because I want to show the steps along the way, for example the dataframe after the data is initially loaded. If speed was the sole goal, I’d use lazy mode to evaluate only once, to get the final results.

Clean up the code

In getting the code working, I often go down paths that don’t work out, and include debugging code. So I remove that code before posting. I also rename functions whose purpose has changed since I originally wrote them, and add docstrings.

Example: During development, I displayed the Polars dataframe to check the results of individual steps. Once those steps were working, I removed that debugging code.

Write a clear narrative

It’s easiest for the reader to understand the post if there’s a clear narrative. So after the code is working, I start writing the narrative. I might start writing a narrative (Markdown text) block, then realize that the narrative would make more sense if I rearranged the code blocks, so I do that and then adjust the narrative.

Example: To find a homologous series, I used four substructure matches. During development, I ran all four at once. But to make the process clearer for the reader, I broke it out into three steps (with one, one, and two substructures, respectively).

Review

Before publishing, clear out the existing results by restarting the Jupyter kernel or closing and reopening the notebook. Then re-run all the code in order to make sure there are no problems when it runs in top-to-bottom order.

Re-read the narrative along with the updated code outputs to make sure the descriptions are still correct.

Publish!

There comes a time when the post is done (enough) and ready to publish. After publishing to a web page, I then review the published page and correct any broken images or links. Finally, I publicize the post on social media such as LinkedIn and Mastodon.

Make any changes after publication

Readers may point out improvements you could make, or the data sources could be updated. Ideally, these can help you accomplish the same results with code that’s shorter or more straightforward.

Example: After I mentioned the discrepancy in a few molecular structures to the maintainers of the optical properties dataset, they updated it. I pulled in the new dataset, which let me simplify the notebook by removing the manual correction of structures.

Why blog

Finally, it’s useful to think about why you want to blog.

Cheminformatics blogging is an opportunity to explore, learn, teach, and connect with others in the field. It’s good experience to recognize when you’ve achieved a cheminformatics objective and are ready to package it up by documenting it with a narrative.

Blogging can build your portfolio, either before you get into the field or after, particularly if the code you write in you your day job isn’t open source. Your blog can be valuable for getting into the field (a post helped me transition into cheminformatics) or establishing your area of interest and expertise.

As a practical benefit to you, your blog can be your personal reference library: I find myself referring to my own blog posts frequently to remember how to do something in cheminformatics because I better remember things that I did myself!

Why some organic molecules have a color: Correlating optical absorption wavelength with conjugated bond chain length

2024-10-15T00:00:00+00:00

Molecules have a color if their electronic energy levels are close enough to absorb visible rather than ultraviolet light. For organic molecules, that’s often because of an extensive chain of conjugated bonds. Can we use cheminformatics to find evidence that increasing conjugated bond chain length decreases absorption wavelength, which makes a molecule colored?

Open this notebook in Google Colab so you can run it without installing anything on your computer

Introduction

Colored molecules have applications in television screens, sensors, and to give color to fabrics, paints, foods, and more. I did my PhD in optical spectroscopy, so I was interested by the open-access database Experimental database of optical properties of organic compounds with more than 20,000 data points. One of the first things that came to mind was that organic colored compounds often get their color from an extensive chain of conjugated bonds. Here are quotes from an online textbook:

If you extend this to compounds with really massive delocalisation, the wavelength absorbed will eventually be high enough to be in the visible region of the spectrum, and the compound will then be seen as colored. A good example of this is the orange plant pigment, beta-carotene - present in carrots, for example.

The more delocalization there is, the smaller the gap between the highest energy pi bonding orbital and the lowest energy pi anti-bonding orbital. To promote an electron therefore takes less energy in beta-carotene than in the cases we’ve looked at so far - because the gap between the levels is less.

Here are two examples of molecules, one cyclic and one acylic, with conjugated pi bonds. In each case, the p orbitals on adjacent atoms line up so that the electrons are delocalized over all the conjugated bonds in the molecule (which is all the bonds in these two molecules).

Attribution: Conjugated Pi Bond Systems from LibreTexts, remixed by Jeremy Monat

The more bonds that electrons can delocalize over, the more pi bonding and anti-bonding orbitals; and the highest occupied molecular orbital (HOMO, the top green line in each diagram below, the highest-energy pi bonding orbital) increases in energy, while the lowest unoccupied molecular orbital (LUMO, the lowest red line in each diagram below, the lowest-energy pi anti-bonding orbital) decreases in energy. The gap between the two becomes smaller, and if the conjugated chain is long enough, the HOMO-LUMO energy gap becomes small enough that it’s in the visible spectrum rather than the ultraviolet. When a molecule absorbs visible light, we perceive that it has a color.

Attribution: What Causes Molecules to Absorb UV and Visible Light from LibreTexts, authored, remixed, and/or curated by Jim Clark.

The visible spectrum starts at about 400 nm (violet) and goes to about 740 nm (red). So a molecule that absorbs light in that range will be perceived as colored.

Attribution: Gringer, Public domain, via Wikimedia Commons.

Cheminformatics exploration

Can we use cheminformatics to find evidence that increasing conjugated bond chain length decreases absorption wavelength? To check, I used the open-access database Experimental database of optical properties of organic compounds from 2020 with >20,000 chromophore-solvent combinations. The optical data can be downloaded as a CSV file (version 3).

Packages setup

import math
from typing import Iterable, Dict, List, Tuple
from IPython.display import display, Math
from functools import cache
import warnings

from PIL import Image
from rdkit import Chem
from rdkit.Chem import AllChem, Draw, Mol
import numpy as np
import polars as pl
import polars.selectors as cs
from polars.exceptions import ColumnNotFoundError
import altair as alt
from great_tables import GT, style, loc
import latexify

# Suppress the specific RDKit IPythonConsole truncation warning for MolsToGridImage
warnings.filterwarnings(
    "ignore", message=r"Truncating the list of molecules to be displayed to \d+"
)

Find longest conjugated bond chain

To find the longest conjugated bond chain in each molecule, we’ll use a graph-traversal algorithm. We’ll define two functions–one to find connected bonds and one to get the longest conjugated bond chain in a molecule–then describe how they work using an example molecule.

def find_connected_bonds(
    graph: np.ndarray,
    start_bond: int,
    visited: Iterable,
) -> List:
    """
    Find connected bonds in an adjacency matrix. Terminology is specific to bonds in molecules, but algorithm should work for any adjacency matrix.

    :param graph: The bond adjacency matrix
    :param start_bond: The bond index to start the chain from
    :param visited: The bond indices already visited; preferably a set for efficiency
    :returns: The list of bonds connected to, and including, the start bond
    """
    # Initialize the stack with the start bond
    stack = [start_bond]
    connected_component = []

    while stack:
        # Pop off the stack the last bond added
        bond_index = stack.pop()
        # Only follow the chain from this bond if this bond hasn't already been visited
        if bond_index not in visited:
            # Add the bond index to the list of visited bonds
            visited.add(bond_index)
            # Note that the bond is connected to the start bond
            connected_component.append(bond_index)

            # Add all neighbors to the stack for traversal if they haven't already been visited
            for neighbor, is_connected in enumerate(graph[bond_index]):
                if is_connected and neighbor not in visited:
                    stack.append(neighbor)
    return connected_component

def get_longest_conjugated_bond_chain(
    mol: Mol, verbose: bool = False
) -> tuple[List[int], np.ndarray]:
    """
    Get the longest conjugated bond chain in a molecule.

    :param mol: The RDKit molecule
    :param verbose: Whether to return the bond_matrix in addition to conjugated_bonds_out
    :returns: The list of bond indices of the longest conjugated bond chain in the molecule; if verbose is true, also the bond adjacency matrix
    """
    # Create a list to store conjugated bond indices
    conjugated_bonds = [
        bond.GetIdx() for bond in mol.GetBonds() if bond.GetIsConjugated()
    ]

    if not conjugated_bonds:
        # No conjugated bonds found, return empty list
        return []

    n_conjugated_bonds = len(conjugated_bonds)

    # Build a subgraph of the conjugated bonds only;
    #   initially populate it with zeroes, indicating bonds are not connected
    bond_matrix = np.zeros((n_conjugated_bonds, n_conjugated_bonds), dtype=int)

    # Populate the bond adjacency matrix for conjugated bonds
    for i, bond_i in enumerate(conjugated_bonds):
        bond_i_obj = mol.GetBondWithIdx(bond_i)
        # Avoid duplicate work by only processing pairs where j < i
        for j, bond_j in enumerate(conjugated_bonds[:i]):
            bond_j_obj = mol.GetBondWithIdx(bond_j)
            # Check if two conjugated bonds share an atom--
            #   do the set of {beginning atom, ending atom} overlap for the two bonds
            if {bond_i_obj.GetBeginAtomIdx(), bond_i_obj.GetEndAtomIdx()} & {
                bond_j_obj.GetBeginAtomIdx(),
                bond_j_obj.GetEndAtomIdx(),
            }:
                # Change the bond matrix value to 1, indicating the two bonds are connected
                bond_matrix[i, j] = 1
                bond_matrix[j, i] = 1

    # Initialize variables to store the longest conjugated bond chain
    visited = set()
    longest_bond_chain = []

    # Starting from each bond, traverse the graph and find the largest connected component
    for start_bond in range(n_conjugated_bonds):
        if start_bond not in visited:
            bond_chain = find_connected_bonds(bond_matrix, start_bond, visited)
            # Note that bonds are added to `visited` in find_connected_bonds(),
            #   so any bonds already visited from a previous starting bond
            #   won't have find_connected_bonds run on it
            # If this chain is longer than the longest one found so far, mark this chain as the longest
            if len(bond_chain) > len(longest_bond_chain):
                longest_bond_chain = bond_chain

    # Convert subgraph bond indices back to the original bond indices
    conjugated_bonds_out = [conjugated_bonds[i] for i in longest_bond_chain]
    conjugated_bonds_out.sort()

    if not verbose:
        return conjugated_bonds_out
    else:
        return conjugated_bonds_out, bond_matrix

Let’s use an example branched molecule to explain how these algorithms work.

C6H8 = Chem.MolFromSmiles("C=CC(=C)C=C")
longest_conjugated_bond_chain, bond_matrix = get_longest_conjugated_bond_chain(
    mol=C6H8, verbose=True
)
Draw.MolToImage(
    C6H8,
    highlightBonds=longest_conjugated_bond_chain,
)

Let’s label the bond indices using RDKit’s addBondIndices.

opts = Draw.MolDrawOptions()
opts.addBondIndices = True
Draw.MolToImage(C6H8,size=(350,300),options=opts)

To understand the bond adjacency matrix, let’s use Polars’ Great Tables integration to make a nicely-formatted table.

# Convert the bond matrix to a Polars DataFrame
df_bond_matrix = pl.DataFrame(bond_matrix)

# Rename columns to add custom labels
df_bond_matrix = df_bond_matrix.rename(
    {f"column_{str(i)}": f"Bond {i}" for i in range(bond_matrix.shape[1])}
)

# Add row index labels (Polars doesn't have an index column, so we'll add it as a new column)
row_indices = [f"Bond {i}" for i in range(bond_matrix.shape[0])]
index_col = "Adjacent?"
df_bond_matrix = df_bond_matrix.insert_column(0, pl.Series(index_col, row_indices))

# Add a Total column to sum up how many bonds a given bond is connected to
total_col = "Total"
df_bond_matrix = df_bond_matrix.with_columns(
    pl.sum_horizontal(cs.starts_with("Bond")).alias(total_col)
)

# Use GreatTables to format the table
GT(df_bond_matrix).tab_options(
    # Bold the column headings
    column_labels_font_weight="bold",
).tab_style(
    # Bold the index and total columns
    style=style.text(weight="bold"),
    locations=loc.body(columns=[index_col, total_col]),
).tab_header(
    # Add a title to the table
    title="Bond adjacency matrix"
)

Adjacent?	Bond 0	Bond 1	Bond 2	Bond 3	Bond 4	Total
Bond adjacency matrix
Bond 0	0	1	0	0	0	1
Bond 1	1	0	1	1	0	3
Bond 2	0	1	0	1	0	2
Bond 3	0	1	1	0	1	3
Bond 4	0	0	0	1	0	1

The table demonstrates which bonds are adjacent, in the sense that the two bonds share an atom. For example, bond 1 is adjacent to bonds 0, 2, and 3. That makes sense based on the molecular diagram.

The bond chain starts with a start bond, in this case 0, and follows all its adjacent bonds to make a chain. Here, the algorithm went to bond 1 (the only bond connected to bond 0), then at the branch chose to go off the molecule’s spine (longest atom chain) to go to bond 2, then followed the other branch to complete the molecule’s spine (bonds 3 and 4):

longest_conjugated_bond_chain

[0, 1, 2, 3, 4]

In this case, a single bond chain found all the conjugated bonds in the molecule. The algorithm loops over all conjugated bonds to make sure it finds the longest chain. But if a given start bond (e.g., 1) was already visited because it was in the bond chain of a previous starting bond (e.g., 0), the algorithm doesn’t re-trace the same bond chain. This is a big computational savings because it avoids unnecessary graph traversals, which are expensive. This is facilitated by the variable visited being passed from get_longest_conjugated_bond_chain() to find_connected_bonds(), where it is modified by adding the nodes visited.

Additional computational savings comes from excluding the non-conjugated bonds from the adjacency matrix. While not noticeable for the example molecule because all its bonds are conjugated, this can greatly reduce the adjacency matrix, and thus the graph traversal, for molecules where some bonds are not conjugated.

Check that longest conjugated bond chain gives expected results

Let’s make sure that our algorithm gives the expected results for a variety of cyclic and acyclic molecules.

examples = {
    "benzene": "c1ccccc1",
    "naphthalene": "c1c2ccccc2ccc1",
    "anthracene": "c1ccc2cc3ccccc3cc2c1",
    "toluene": "c1ccccc1C",
    "benzene + 5 linear conjugated bonds": "c1ccccc1CC=CC=CC=C",
    "benzene + 7 linear conjugated bonds": "c1ccccc1CC=CC=CC=CC=C",
    "1,3-butadiene": "C=CC=C",
    "branched": "C=C\C=C/C(/C=C)=C/C=C",
    "branched + distant": "C=C\C=C\C\C=C\C=C/C(/C=C)=C/C=C",
}
mols = [Chem.MolFromSmiles(sml) for sml in examples.values()]
conjugated_bonds = [get_longest_conjugated_bond_chain(mol) for mol in mols]
Draw.MolsToGridImage(
    mols=mols,
    legends=examples.keys(),
    highlightBondLists=conjugated_bonds,
    subImgSize=(300, 200),
)

Those highlighted conjugated bond chains are as expected, for example

all the bonds are conjugated in benzene, as well as fused polycyclic molecules naphthalene and anthracene
the bond off the ring in toluene is not conjugated
if we put a side-chain on benzene with fewer than six conjugated bonds, the benzyl moiety remains the longest conjugated chain; but if we have seven conjugated bonds on the side chain, it becomes the longest conjugated chain
in the “branched + distant” molecule, if we break the conjugation chain by having two C-C single bonds in a row, the chain does not include the distant, disconnected conjugated bonds

Of course we should check beta-carotene, whose color comes from its extended conjugated bond chain and is “responsible for the orange color of carrots”. beta-carotene “absorbs most strongly between 400-500 nm. This is the green/blue part of the spectrum.” Because it absorbs those wavelengths, when we look at a sample of beta-carotene we see the reflected light of the complimentary colors, so we perceive an orange color. Note that beta-carotene’s absorption isn’t that much lower energy (higher wavelength) than ultraviolet (which goes up to 400 nm), so molecules with longer conjugated chains might absorb at higher wavelengths such as 565-590 nm (yellow), giving them perceived colors towards blue, magenta, and purple.

beta_carotene = Chem.MolFromSmiles(
    "CC2(C)CCCC(\C)=C2\C=C\C(\C)=C\C=C\C(\C)=C\C=C\C=C(/C)\C=C\C=C(/C)\C=C\C1=C(/C)CCCC1(C)C"
)
longest_conjugated_bond_chain = get_longest_conjugated_bond_chain(beta_carotene)
print(
    f"beta-carotene's longest conjugated bond chain is {len(longest_conjugated_bond_chain)} bonds long."
)
Draw.MolToImage(
    beta_carotene,
    highlightBonds=longest_conjugated_bond_chain,
    size=(800, 200),
)

beta-carotene's longest conjugated bond chain is 21 bonds long.

And beta-carotene’s molecular structure is also beautifully symmetric.

Now that we have an algorithm to find the longest conjugated bond chain in a molecule, let’s apply it to the optical dataset.

Prepare data

Let’s read into a Polars dataframe the data from the optical dataset CSV file.

df = pl.read_csv("../data/DB for chromophore_Sci_Data_rev03.csv")

# If there are any rows where all values are null, drop those rows
df = df.filter(~pl.all_horizontal(pl.all().is_null()))  

df

shape: (20_836, 14)

Tag	Chromophore	Solvent	Absorption max (nm)	Emission max (nm)	Lifetime (ns)	Quantum yield	log(e/mol-1 dm3 cm-1)	abs FWHM (cm-1)	emi FWHM (cm-1)	abs FWHM (nm)	emi FWHM (nm)	Molecular weight (g mol-1)	Reference
i64	str	str	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	str
0	"N#Cc1cc2ccc(O)cc2oc1=O"	"O"	355.0	410.0	2.804262	null	null	null	null	null	null	187.026943	"10.1021/acs.jpcb.5b09905"
1	"N#Cc1cc2ccc([O-])cc2oc1=O"	"O"	408.0	450.0	3.961965	null	null	null	2128.3	null	43.2	186.019667	"10.1021/acs.jpcb.5b09905"
2	"CCCCCCCCCCCC#CC#CCCCCCCCCCN1C(…	"ClC(Cl)Cl"	526.0	535.0	3.602954	null	null	null	null	null	null	1060.705709	"10.1002/smll.201901342"
3	"[O-]c1c(-c2nc3ccccc3s2)cc2ccc3…	"CC#N"	514.0	553.7	3.81	null	null	null	2120.5	null	65.2	350.064509	"10.1016/j.snb.2018.10.043"
4	"[O-]c1c(-c2nc3ccccc3s2)cc2ccc3…	"CS(C)=O"	524.0	555.0	4.7	null	null	2219.7	1565.6	61.2	48.3	350.064509	"10.1016/j.snb.2018.10.043"
…	…	…	…	…	…	…	…	…	…	…	…	…	…
20831	"N#Cc1c(N2CCCCC2)cc(-c2ccc3ccc4…	"C1CCOC1"	344.0	473.0	null	null	null	null	3937.1	null	88.9	488.188863	"10.1021/ol9000679"
20832	"CCCCCCn1c2c(c3ccccc31)-c1c(c3c…	"ClCCl"	312.0	352.0	null	null	4.460898	null	null	null	null	797.18392	"DOI: 10.1021/ol501083d"
20833	"CCCCCCn1c2c(c3cc(-c4ccc5ccc6cc…	"ClCCl"	340.0	382.0	null	null	4.70757	null	null	null	null	1598.142	"DOI: 10.1021/ol501083d"
20834	"CCCCCCn1c2c(c3ccccc31)-c1c(c3c…	"CCCCCCn1c2c(c3ccccc31)-c1c(c3c…	321.0	494.0	null	null	null	null	null	null	null	797.18392	"DOI: 10.1021/ol501083d"
20835	"CCCCCCn1c2c(c3cc(-c4ccc5ccc6cc…	"CCCCCCn1c2c(c3cc(-c4ccc5ccc6cc…	365.0	466.0	null	null	null	null	null	null	null	1598.142	"DOI: 10.1021/ol501083d"

The dataset gives absorption and emission maxima as wavelengths. That makes sense because spectroscopists wavelength describes the color of the light used in the laboratory. But to compare, for example, the absorption and emission maximum for a molecule, it’s better to use energy units to express the difference between different molecular energy levels. So let’s convert wavelengths to energies in electron volts, eV.

To determine the conversion factor, let’s use the equation for energy E as a function of wavelength λ.

@latexify.function
def E(
    h: float,
    c: float,
    λ: float,
) -> float:
    """Calculate the binomial coefficient: how many ways there are to choose k items from n items.

    :param h: Planck's constant
    :param c: speed of light
    :param λ: wavelength
    :returns: energy
    """
    return h * c / λ


E

\[\displaystyle E(h, c, λ) = \frac{h c}{λ}\]

Let’s plug in the values of the physical constants to four decimal places and use factor-label dimensional analysis:

eqn = r"$E = \frac{6.6261 \times 10^{-34} \, \text{J} \cdot \text{s} \cdot 2.9979 \times 10^8 \, \text{m/s}}{\lambda \times 10^{-9} \, \text{m}} \cdot \frac{1 \, \text{eV}}{1.6022 \times 10^{-19} \, \text{J}} = \frac{1239.8 \, \text{eV}}{\lambda}$"
display(Math(eqn))

$\displaystyle E = \frac{6.6261 \times 10^{-34} \, \text{J} \cdot \text{s} \cdot 2.9979 \times 10^8 \, \text{m/s}}{\lambda \times 10^{-9} \, \text{m}} \cdot \frac{1 \, \text{eV}}{1.6022 \times 10^{-19} \, \text{J}} = \frac{1239.8 \, \text{eV}}{\lambda}$

Doing that in Python to store the value in a variable:

h = 6.6261e-34  # J*s
c = 2.9979e8  # m/s
nm = 1e-9  # m
eV = 1.6022e-19  # J
eV_nm = h * c / (nm * eV)
eV_nm

1239.8193228061414

Now we can convert absorption and emission maxima to energy units of eV, then calculate their difference as the Stokes shift. The Stokes shift reflects how much the molecule relaxes from its initial excited state (the Franck–Condon state) to the lowest vibrational level in the excited state that it typically emits light from. In the following diagram, the blue arrow represents absorption from the ground state to the Franck-Condon state, and the green arrow represents emission from the relaxed excited state back to the ground state. The blue arrow is longer, representing the greater energy of absorption. The difference between the vertical length of the blue and green arrows is the Stokes shift.

Attribution: Franck Condon Diagram on Wikipedia by Samoza, licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license

# To prevent duplicate-column errors when re-running this code, drop the columns we're about to add
for column in [
    "longest_bond_indices",
    "Longest conjugated bond length",
    "Absorption max (eV)",
    "Emission max (eV)",
    "Stokes shift (eV)",
]:
    try:
        df.drop(column)
    except ColumnNotFoundError:
        pass

Now we can calculate the energies in eV from the wavelengths in nm.

df = df.with_columns(
    [
        (eV_nm / pl.col("Absorption max (nm)")).alias("Absorption max (eV)"),
        (eV_nm / pl.col("Emission max (nm)")).alias("Emission max (eV)"),
    ]
).with_columns(
    (pl.col("Absorption max (eV)") - pl.col("Emission max (eV)")).alias(
        "Stokes shift (eV)"
    ),
)

Finding the longest conjugated chain for each molecule

Now we come to the computationally-intensive operation: Finding the the longest conjugated chain for each molecule using conjugated_chain() which calls get_longest_conjugated_bond_chain(). We define a function that will return the indices and length of the longest conjugated bond chain. The data set repeats some chromophores: there are a little less than three rows per chromophore. So we’ll cache the results for each chromophore to avoid recalculating for each of its rows. Python’s built-in module functools include a cache decorator that makes this simple.

@cache
def conjugated_chain(sml) -> Dict[str : List[int], str:int]:
    """
    Find the indices and length for the longest bond chain in a SMILES.

    :param sml: SMILES to be made into a molecule
    :returns: A dictionary of longest bond chain indices and longest bond chain length
    """
    return_dict = dict()
    mol = Chem.MolFromSmiles(sml)
    longest_bond_indices = get_longest_conjugated_bond_chain(mol)
    return_dict["longest_bond_indices"] = longest_bond_indices
    return_dict["Longest conjugated bond length"] = len(longest_bond_indices)
    return return_dict

Now we use Polars’ map_elements to calculate the longest bond chain for each molecule. Because conjugated_chain() returns a dictionary, Polars treats it as a struct, which we can then unnest to create a column for each dictionary key-value pair. We then sort the dataframe to put the longest bond chain lengths first so we can examine those molecules. Finally, we use Polars’ shrink_to_fit() to decrease the dataframe memory usage and prevent problems with plotting.

This entire operation takes about 13 seconds on my laptop with caching, and about 33 seconds without, which roughly reflects the ratio of unique chromophores to data points. Caching is thus demonstrated to be effective here.

# This may take 12-60 seconds: Finding the longest conjugated chain in each molecule in the dataframe
df = (
    df.with_columns(
        conjugated=pl.col("Chromophore").map_elements(
            lambda sml: conjugated_chain(sml), return_dtype=pl.Struct
        )
    )
    .unnest("conjugated")
    .sort(pl.col("Longest conjugated bond length"), descending=True)
    .shrink_to_fit()
)

df.head()

shape: (5, 19)

Tag	Chromophore	Solvent	Absorption max (nm)	Emission max (nm)	Lifetime (ns)	Quantum yield	log(e/mol-1 dm3 cm-1)	abs FWHM (cm-1)	emi FWHM (cm-1)	abs FWHM (nm)	emi FWHM (nm)	Molecular weight (g mol-1)	Reference	Absorption max (eV)	Emission max (eV)	Stokes shift (eV)	longest_bond_indices	Longest conjugated bond length
i64	str	str	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	str	f64	f64	f64	list[i64]	i64
16578	"N#CC(C#N)=C1C=C(C=Cc2ccc(-c3cc…	"N#CC(C#N)=C1C=C(C=Cc2ccc(-c3cc…	null	706.0	null	0.12	null	null	null	null	null	2731.043503	"10.1039/c6tc03359h"	null	1.756118	null	[0, 1, … 242]	243
16648	"C(=Cc1ccc(C=Cc2ccc(N(c3ccc(-c4…	"C(=Cc1ccc(C=Cc2ccc(N(c3ccc(-c4…	null	660.0	null	null	null	null	null	null	null	1994.81382	"10.1039/c6tc03359h"	null	1.878514	null	[0, 1, … 177]	178
16555	"CC(C)c1cccc(C(C)C)c1N1C(=O)c2c…	"CC(C)c1cccc(C(C)C)c1N1C(=O)c2c…	null	790.0	null	null	null	null	null	null	null	2094.857519	"10.1039/c6tc03359h"	null	1.569392	null	[3, 4, … 185]	174
16554	"CC(C)c1cccc(C(C)C)c1N1C(=O)c2c…	"CC(C)c1cccc(C(C)C)c1N1C(=O)c2c…	null	860.0	null	null	null	null	null	null	null	2030.87786	"10.1039/c6tc03359h"	null	1.44165	null	[3, 4, … 181]	170
15298	"c1ccc(C(=C(c2ccccc2)c2ccc(-c3c…	"Cc1ccccc1"	530.0	637.0	null	0.252	null	null	null	null	null	1760.698122	"10.1016/j.dyepig.2012.08.028"	2.339282	1.946341	0.392941	[0, 1, … 157]	158

Checking the results, we find that the longest conjugated bond chain length in the optical dataset is 158 bonds! That’s more than seven times longer than beta-carotene’s. So it certainly makes sense that some of these molecules absorb visible light. For example, the molecule with the longest conjugated bond chain has its absorption maximum at 530 nm in the solvent of toluene.

Checking for a correlation of absorption wavelength against longest conjugated bond chain length

Let’s use Polars’ plot capability to plot absorption max against longest bond length to check for any trends. Altair can plot a maximum of 5,000 data points, and there are more than 20,000 points in the optical dataset. We could enable VegaFusion to allow for more points, but that would be an additional dependency and it seems not to work well with Google Colab. Instead, let’s filter down to the ~7k unique chromophores in the dataset and plot the first 5k. Polars will select one solvent essentially at random for each chromophore, and the solvatochromic shift is generally not huge compared to the range we’ll be plotting (from <200 to 850 nm), so that’s a reasonable sampling. And 5k points should be plenty to discern if there’s a trend.

df_unique_chromophore = df.unique(subset="Chromophore")

We also need to drop the column longest_bond_indices because large lists of integers are not supported (we’re not plotting them anyway)

Now we can make a plot of absorption maximum against longest conjugated bond length for the first 5k unique chromophores.

df_unique_chromophore.slice(0, 5000).drop("longest_bond_indices").plot.scatter(
    x="Longest conjugated bond length", y="Absorption max (nm)"
)

There’s not much of a correlation, presumably because of the varied molecular structures.

Let’s remove other variables by finding a series of molecules with a similar structure, but where the longest conjugated bond chain length increases. Given our dataset, let’s start with molecules with a large number of connected conjugated bonds, then hopefully find a simple, linear molecule with a repeat unit that we can look for molecular analogues with fewer repeat units. We’ll plot the top 50 molecules, which is the RDKit’s default maximum for MolsToGridImage.

# Filter down to unique chromophores to avoid repeats in the molecular grid
df_unique_chromophore = df_unique_chromophore.sort(
    "Longest conjugated bond length", descending=True
)

# Extract columns as lists so we can plot molecules in a grid
unique_chromophores = df_unique_chromophore["Chromophore"].to_list()
tags = df_unique_chromophore["Tag"].to_list()
longest_conjugated_bond_lengths = df_unique_chromophore["Longest conjugated bond length"].to_list()
legends = [
    f"{longest_bond_length} bonds: tag {tag}"
    for (tag, longest_bond_length) in zip(tags, longest_conjugated_bond_lengths)
]
mols = [Chem.MolFromSmiles(chromophore) for chromophore in unique_chromophores]
Draw.MolsToGridImage(
    mols=mols,
    legends=legends,
    molsPerRow=5,
)

While there are a lot of interesting-looking molecules, tag 19940 has a simple linear structure with a clear repeat unit that includes an anthracene moiety. This molecule has six of those repeat units, so we’ll refer to it as the n = 6 molecule. Let’s show the molecule with its conjugated chain highlighted.

size_wide_mol = (1000, 200)
df_n6 = df_unique_chromophore.filter(Tag=19940)
sml_n6 = df_n6[0]["Chromophore"].item()
mol_n6 = Chem.MolFromSmiles(sml_n6)
tags = df_n6[0]["Tag"].item()
longest_bond_length = df_n6[0]["Longest conjugated bond length"].item()
legend = f"{longest_bond_length} bonds: tag {tags}"
conjugated_bonds_n6 = df_n6[0]["longest_bond_indices"].item().to_list()
Draw.MolToImage(
    mol=mol_n6,
    size=size_wide_mol,
    legend=legend,
    highlightBonds=conjugated_bonds_n6,
)

By the way, sometimes it’s difficult to see that all the specified bonds have been highlighted in the molecular image; there is in fact a contiguous chain from the first triple bond to the last.

Now let’s show the repeat unit in the same orientation as the molecule, which requires rotating the atoms of the repeat unit by 90 degrees. We’ll rotate serval molecules in this post by 90 degrees, so let’s define a function to do that.

def rotate_90(sml: str) -> Mol:
    """
    Rotates a molecule by 90 degrees based on its SMILES string and visualizes it.

    :param sml: SMILES string of the molecule to rotate.
    :returns: The rotated molecule.
    """

    mol = Chem.MolFromSmiles(sml)

    # Generate 2D coordinates if not already present
    AllChem.Compute2DCoords(mol)

    # Define the rotation matrix for 90 degrees (π/2 radians)
    theta = np.pi / 2  # 90 degrees in radians
    rotation_matrix = np.array(
        [[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]]
    )

    # Get the conformation of the molecule (the 2D coordinates)
    conf = mol.GetConformer()

    # Apply the rotation to each atom
    for i in range(mol.GetNumAtoms()):
        # Get the current x, y, z coordinates (z will stay 0.0 for 2D)
        pos = conf.GetAtomPosition(i)
        x, y = pos.x, pos.y  # Extract x, y coordinates

        # Apply the 90-degree rotation
        new_x, new_y = np.dot(rotation_matrix, [x, y])

        # Set the new coordinates, keeping z=0.0
        conf.SetAtomPosition(i, (new_x, new_y, 0.0))

    # Return the rotated molecule
    return mol

Now we can show the repeat unit in the same orientation that it appears in the n = 6 molecule. The RDKit tends to recalculate molecular coordinates, so to be safe we assign the output image of Draw.MolToImage to a variable, then display that image.

repeat_unit = "CC1=C2C=CC=CC2=C(C#C)C2=CC=CC=C12"
repeat_unit_mol = Chem.MolFromSmiles(repeat_unit)

repeat_unit_mol = rotate_90(repeat_unit)
image = Draw.MolToImage(
    repeat_unit_mol,
)
display(image)

Let’s check the dataframe for molecules with the same overall structure, with fewer repeat units. To do that, we’ll need to check for a substructure in a molecule.

Filter to molecules in the series

Let’s define a function to check how many of a given substructure there are in a molecule.

def match_counts(
    sml: str,
    smls_to_match: Iterable[str] = None,
) -> Dict[str, int]:
    """
    Convert target and to-match SMILES to RDKit molecules, then count how many times the to-match molecules occur in the target molecules.

    :param sml: SMILES to convert to a molecule
    :param smls_to_match: One or more SMILES to find as a substructure
    :returns: Dictionary with name:count entry for each SMILES to match
    """
    mol = Chem.MolFromSmiles(sml)
    return_dict = dict()
    for name, sml_to_match in smls_to_match.items():
        mol_to_match = Chem.MolFromSmiles(sml_to_match)
        matches = mol.GetSubstructMatches(mol_to_match)
        return_dict[f"{name}_match_count"] = len(matches)
    return return_dict

To prevent double-counting of the number of repeat units, we use a substructure one atom longer than we showed above.

smls_to_match = {
    # Repeat unit containing an anthracene moiety, with two triple bonds
    "repeat_unit": "C#CC1=C2C=CC=CC2=C(C#C)C2=CC=CC=C12",
}

Let’s highlight one instance of the repeat unit, and of the terminal group, in the n = 6 molecule. Because we’ll search by atoms (the substructure), and then want to highlight the bonds as well, we’ll define a function to get the bond indices connecting an iterable of atoms.

def get_bond_indices_connecting_atoms(
    mol: Mol, atom_indices: Iterable[int]
) -> List[int]:
    """
    Given an RDKit molecule and a list of atom indices, return the bond indices
    that connect the atoms in the list.

    :param mol: RDKit molecule object
    :param atom_indices: List of atom indices to check for connectivity
    :return: List of bond indices that connect the atoms in atom_indices
    """
    bond_indices = []
    atom_set = set(atom_indices)  # Using a set for faster lookup

    # Iterate through all bonds in the molecule
    for bond in mol.GetBonds():
        begin_atom = bond.GetBeginAtomIdx()
        end_atom = bond.GetEndAtomIdx()

        # Check if both atoms of the bond are in the provided list of atom indices
        if begin_atom in atom_set and end_atom in atom_set:
            bond_indices.append(bond.GetIdx())

    return bond_indices

# Get the first repeat unit substructure match
match_repeat_unit = mol_n6.GetSubstructMatches(repeat_unit_mol)[0]
bond_indices_repeat_unit = get_bond_indices_connecting_atoms(mol_n6, match_repeat_unit)
Draw.MolToImage(
    mol_n6,
    highlightAtoms=match_repeat_unit,
    highlightBonds=bond_indices_repeat_unit,
    size=size_wide_mol,
)

To avoid duplicate-column errors when the code is run more than once, we’ll drop any existing match count columns. Then we’ll count the occurrences of each substructure in each molecule.

def add_match_counts(
    df: pl.DataFrame,
    smls_to_match: Iterable[str],
) -> pl.DataFrame:
    """
    For a dataframe with SMILES, add the count of matching substructures for SMILES to match.

    :param df: Polars dataframe of molecules with SMILES in Chromophore column
    :param smls_to_match: Iterable of SMILES for substructures
    :returns: Polars dataframe with match count columns added
    """
    df = df.drop(cs.ends_with("_match_count"))
    df = (
        df.with_columns(
            substructure_counts=pl.col("Chromophore").map_elements(
                lambda sml: match_counts(sml, smls_to_match), return_dtype=pl.Struct
            )
        )
        .unnest("substructure_counts")
        .sort("Longest conjugated bond length", descending=True)
        .shrink_to_fit()
    )
    return df

df = add_match_counts(
    df=df,
    smls_to_match=smls_to_match,
)

Let’s filter down to molecules that contain the repeat unit by requiring the repeat unit match count be greater than zero.

df.filter((pl.col("repeat_unit_match_count") > 0)).sort(
    pl.col("repeat_unit_match_count")
)

shape: (53, 20)

Tag	Chromophore	Solvent	Absorption max (nm)	Emission max (nm)	Lifetime (ns)	Quantum yield	log(e/mol-1 dm3 cm-1)	abs FWHM (cm-1)	emi FWHM (cm-1)	abs FWHM (nm)	emi FWHM (nm)	Molecular weight (g mol-1)	Reference	Absorption max (eV)	Emission max (eV)	Stokes shift (eV)	longest_bond_indices	Longest conjugated bond length	repeat_unit_match_count
i64	str	str	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	str	f64	f64	f64	list[i64]	i64	i64
19379	"CCCCCCCCn1c(=O)c2cc3c(C#Cc4ccc…	"ClC(Cl)Cl"	582.0	641.0	3.4	0.53	4.518514	null	1387.6	null	57.1	1074.508407	"10.1021/acs.joc.6b00364"	2.130274	1.934196	0.196078	[8, 9, … 91]	76	1
19384	"CCCCCCCCn1c(=O)c2cc3c(C#Cc4ccc…	"c1ccccc1"	584.0	627.0	null	null	null	null	null	null	null	1074.508407	"10.1021/acs.joc.6b00364"	2.122978	1.977383	0.145595	[8, 9, … 91]	76	1
19388	"CCCCCCCCn1c(=O)c2cc3c(C#Cc4ccc…	"C1CCOC1"	563.0	641.0	null	null	null	null	null	null	null	1074.508407	"10.1021/acs.joc.6b00364"	2.202166	1.934196	0.26797	[8, 9, … 91]	76	1
20066	"O=c1c2ccccc2c2nc3cc4c(C#C[Si](…	"ClCCl"	731.0	null	null	null	null	null	null	null	null	1238.628931	"10.1021/acs.joc.9b02756"	1.696059	null	null	[0, 1, … 107]	64	1
20073	"O=c1c2ccccc2c2nc3cc4c(C#C[Si](…	"O=c1c2ccccc2c2nc3cc4c(C#C[Si](…	743.0	null	null	null	null	null	null	null	null	1238.628931	"10.1021/acs.joc.9b02756"	1.668667	null	null	[0, 1, … 107]	64	1
…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…
19936	"CC(C)[Si](C#Cc1c2ccc(C(C)(C)C)…	"Cc1ccccc1"	523.0	541.0	null	0.02	4.694605	3158.9	1770.3	87.0	51.9	962.658106	"10.1021/acs.joc.8b00311"	2.370591	2.291718	0.078874	[4, 5, … 74]	39	2
19937	"CC(C)[Si](C#Cc1c2ccc(C(C)(C)C)…	"Cc1ccccc1"	570.0	589.0	null	0.022	4.781037	3541.0	911.6	116.2	31.6	1274.845907	"10.1021/acs.joc.8b00311"	2.175122	2.104956	0.070165	[4, 5, … 101]	58	3
19938	"CC(C)[Si](C#Cc1c2ccc(C(C)(C)C)…	"Cc1ccccc1"	582.0	609.0	null	0.02	4.975432	3895.7	1206.4	133.7	44.8	1587.033707	"10.1021/acs.joc.8b00311"	2.130274	2.035828	0.094446	[4, 5, … 128]	77	4
19939	"CC(C)[Si](C#Cc1c2ccc(C(C)(C)C)…	"Cc1ccccc1"	585.0	623.0	null	0.018	5.130334	3751.8	1204.5	130.0	46.8	1899.221508	"10.1021/acs.joc.8b00311"	2.119349	1.990079	0.12927	[4, 5, … 155]	96	5
19940	"CC(C)[Si](C#Cc1c2ccc(C(C)(C)C)…	"Cc1ccccc1"	589.0	629.0	null	null	5.178977	3681.3	1318.7	129.2	52.3	2211.409309	"10.1021/acs.joc.8b00311"	2.104956	1.971096	0.133861	[4, 5, … 182]	115	6

That’s too many molecules–we expect as many as six, for n = 1-6. Let’s examine the molecular structures to understand why.

Because we’ll plot several instances of molecular grids from subsets of the dataframe, we define a function to plot first ten molecules and legends based on the columns in the dataframe.

def df_to_grid_image(
    df: pl.DataFrame,
    subImgSize: Tuple[int, int] = (200, 200),
    molsPerRow=3,
) -> Image.Image:
    """
    Create a molecular grid image from a Polars dataframe.

    :param df: Polars dataframe
    :param subImgSize: The size for each molecule
    :param molsPerRow: The number of molecules in each row
    :returns: Molecular grid image
    """
    # Fill any null or NaN (not a number) values to prevent errors during processing
    df = df.fill_null(0).fill_nan(0)

    matching = df["Chromophore"].to_list()
    mols = [Chem.MolFromSmiles(match) for match in matching]
    nums_match = df["repeat_unit_match_count"].to_list()
    absorbances_nm = df["Absorption max (nm)"].to_list()
    tags = df["Tag"].to_list()
    legends = []
    for index, num_match in enumerate(nums_match):
        this_tag = tags[index]
        legend = f"Tag {this_tag}: {num_match} unit(s)"
        abs_nm = absorbances_nm[index]
        if not math.isnan(abs_nm):
            legend += f" {abs_nm}nm"
        legends.append(legend)
    conjugated_bonds = [get_longest_conjugated_bond_chain(mol) for mol in mols]
    print(f"{len(mols)} molecules")

    dwg = Draw.MolsToGridImage(
        mols=mols,
        legends=legends,
        maxMols=10,
        molsPerRow=molsPerRow,
        highlightBondLists=conjugated_bonds,
        subImgSize=subImgSize,
    )
    return dwg

df_to_grid_image(
    df.filter((pl.col("repeat_unit_match_count") > 0)).sort(
        pl.col("repeat_unit_match_count")
    )
)

53 molecules

None of the first 10 molecules have the two terminal triisopropylsyl groups. Let’s specify that we want two such terminal groups by adding the terminal group to the list of SMILES to match, then filtering to molecules with two of those terminal groups.

smls_to_match = smls_to_match | {
    # Terminal triisopropylsyl group
    "terminal": "CC(C)[SiH](C(C)C)C(C)C"
}

Here are the substructures we’re searching for now:

mols_to_match = []
for sml in smls_to_match.values():
    mol = rotate_90(sml)
    mols_to_match.append(mol)
image = Draw.MolsToGridImage(
    mols=mols_to_match,
    legends=smls_to_match.keys(),
    molsPerRow=4,
    subImgSize=(300, 350),
)
display(image)

df = add_match_counts(
    df=df,
    smls_to_match=smls_to_match,
)

df_to_grid_image(
    df.filter(
        (pl.col("repeat_unit_match_count") > 0) & (pl.col("terminal_match_count") == 2)
    ).sort(pl.col("repeat_unit_match_count"))
)

19 molecules

That helps, but we’re getting a lot of fused-ring systems with more than three rings. Let’s exclude those by adding SMILES to match that contain four fused rings, where the fourth ring is either all carbons, or contains two nitrogens. We’ll then filter to molecules whose match count is zero four-fused-ring substructures.

smls_to_match = smls_to_match | {
    # Four benzene rings
    "too_many_rings": "C#CC1=C2C=C3C=CC=CC3=CC2=C(C#C)C2=CC=CC=C12",
    # Four rings: Three benzene and one dinitrogen ring
    "too_many_rings_N": "C#CC1=C2C=C3N=CC=NC3=CC2=C(C#C)C2=CC=CC=C12",
}

Here are all the substructures we’re searching for now, where the first row are substructures we want to include, while the second row is substructures we want to exclude.

mols_to_match = []
for sml in smls_to_match.values():
    mol = rotate_90(sml)
    mols_to_match.append(mol)
image = Draw.MolsToGridImage(
    mols=mols_to_match,
    legends=smls_to_match.keys(),
    molsPerRow=2,
    subImgSize=(300, 350),
)
display(image)

df = add_match_counts(
    df=df,
    smls_to_match=smls_to_match,
)

Next we exclude the four-fused-ring substructures by setting their count to zero.

df_matches = df.filter(
    (pl.col("repeat_unit_match_count") > 0)
    & (pl.col("terminal_match_count") == 2)
    & (pl.col("too_many_rings_match_count") == 0)
    & (pl.col("too_many_rings_N_match_count") == 0)
).sort(pl.col("repeat_unit_match_count"))
df_to_grid_image(
    df_matches,
    subImgSize=(900, 300),
    molsPerRow=3,
)

6 molecules

df_matches

shape: (6, 23)

Tag	Chromophore	Solvent	Absorption max (nm)	Emission max (nm)	Lifetime (ns)	Quantum yield	log(e/mol-1 dm3 cm-1)	abs FWHM (cm-1)	emi FWHM (cm-1)	abs FWHM (nm)	emi FWHM (nm)	Molecular weight (g mol-1)	Reference	Absorption max (eV)	Emission max (eV)	Stokes shift (eV)	longest_bond_indices	Longest conjugated bond length	repeat_unit_match_count	terminal_match_count	too_many_rings_match_count	too_many_rings_N_match_count
i64	str	str	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	str	f64	f64	f64	list[i64]	i64	i64	i64	i64	i64
19935	"CC(C)[Si](C#Cc1c2ccc(C(C)(C)C)…	"Cc1ccccc1"	440.0	461.0	null	0.92	4.4133	2192.4	1449.0	42.5	30.8	650.470305	"10.1021/acs.joc.8b00311"	2.817771	2.689413	0.128358	[4, 5, … 47]	20	1	2	0	0
19936	"CC(C)[Si](C#Cc1c2ccc(C(C)(C)C)…	"Cc1ccccc1"	523.0	541.0	null	0.02	4.694605	3158.9	1770.3	87.0	51.9	962.658106	"10.1021/acs.joc.8b00311"	2.370591	2.291718	0.078874	[4, 5, … 74]	39	2	2	0	0
19937	"CC(C)[Si](C#Cc1c2ccc(C(C)(C)C)…	"Cc1ccccc1"	570.0	589.0	null	0.022	4.781037	3541.0	911.6	116.2	31.6	1274.845907	"10.1021/acs.joc.8b00311"	2.175122	2.104956	0.070165	[4, 5, … 101]	58	3	2	0	0
19938	"CC(C)[Si](C#Cc1c2ccc(C(C)(C)C)…	"Cc1ccccc1"	582.0	609.0	null	0.02	4.975432	3895.7	1206.4	133.7	44.8	1587.033707	"10.1021/acs.joc.8b00311"	2.130274	2.035828	0.094446	[4, 5, … 128]	77	4	2	0	0
19939	"CC(C)[Si](C#Cc1c2ccc(C(C)(C)C)…	"Cc1ccccc1"	585.0	623.0	null	0.018	5.130334	3751.8	1204.5	130.0	46.8	1899.221508	"10.1021/acs.joc.8b00311"	2.119349	1.990079	0.12927	[4, 5, … 155]	96	5	2	0	0
19940	"CC(C)[Si](C#Cc1c2ccc(C(C)(C)C)…	"Cc1ccccc1"	589.0	629.0	null	null	5.178977	3681.3	1318.7	129.2	52.3	2211.409309	"10.1021/acs.joc.8b00311"	2.104956	1.971096	0.133861	[4, 5, … 182]	115	6	2	0	0

Great! We have just the molecules we want, and the dataset contains all six oligomers with the anthracene repeat unit: n = 1-6.

By the way, those four substructure criteria are sufficient to get just the set of molecules we want from this dataset. If we had a larger dataset, we might have to add more criteria to exclude other molecules.

Also, the solvent is toluene in all cases, meaning there are no different solvent effects to consider.

Chem.MolFromSmiles(df_matches[0]["Solvent"].item())

It turns out these molecules all come from one paper, “Synthesis and Electronic Properties of Length-Defined 9,10-Anthrylene−Butadiynylene Oligomers” by Nagaoka et al., DOI 10.1021/acs.joc.8b00311. They actually made this series of molecules for their optical properties: “These oligomers will be attractive as core units in the molecular design of linear π-conjugated compounds for functional dyes and electronic devices.” Their calculated Stokes shift are the same as in our table above.

They also note that the “reaction mixture instantly turned deep purple as a result of the formation of a mixture of oligomers.” This is in agreement with our earlier statement about complimentary colors: Because the n = 1-6 oligomers absorb from 440-589 nm, the complimentary colors will include purple.

As a side note, from examining the dataframe, the photoluminescence quantum yield is 0.92 for n = 1, and about 0.02 for n = 2-5 (a value is not provided for n = 6). Nagaoka et al. attribute this to the fact that “the presence of diacetylene moieties facilitates quenching via nonradiative pathways, as reported for other butadiyne compounds.”

Plot results to check for correlation between color and conjugation chain length

Now that we have a consistent series of molecules, let’s plot their optical properties to check for trends. We’ll use the altair library that provides Polars’ plotting capability.

We start by setting our shape scheme for the absorption, emission, and Stokes shift series.

emission_shape = "circle"
absorption_shape = "triangle"
stokes_shape = "square"

Next we calculate plot ranges based on the data. For all axes, we’ll allow for a slight buffer so our data points aren’t on an edge of the plot.

# Calculate the minimum and maximum y-value in nm across the absorption and emission series
min_y_value_nm = min(
    df_matches["Absorption max (nm)"].min(), df_matches["Emission max (nm)"].min()
)
y_min_nm = min_y_value_nm * 0.95
max_y_value_nm = max(
    df_matches["Absorption max (nm)"].max(), df_matches["Emission max (nm)"].max()
)
y_max_nm = max_y_value_nm * 1.05

# Calculate the minimum and maximum y-value in eV across the absorption and emission series
min_y_value_eV = min(
    df_matches["Absorption max (eV)"].min(), df_matches["Emission max (eV)"].min()
)
y_min_eV = min_y_value_eV * 0.95
max_y_value_eV = max(
    df_matches["Absorption max (eV)"].max(), df_matches["Emission max (eV)"].max()
)
y_max_eV = max_y_value_eV * 1.05

# Calculate the maximum y-value in eV for the Stokes shifts
stokes_y_max = df_matches["Stokes shift (eV)"].max() * 1.1

# Calculate the maximum x-value for all series, based on the maximum repeat unit count
max_repeat_count = df_matches["repeat_unit_match_count"].max() + 0.5

Now we can plot the data. Let’s start with the wavelength units in the source dataset, namely nanometers (nm). We’ll make a scatter plot for the absorption series, then one for the emission series, and layer them on top of each other. To emphasize the color of the light, let’s make each symbol the approximate color of that light, for example 440 nm is violet. For the legend, we’ll make a separate “plot” and concatenate it horizontally with the main plot.

# Order of curves from top to bottom
shapes = [emission_shape, absorption_shape]

# Larger data point size for better visibility
point_size = 150

# Thickness of black outline on data points
stroke_width = 2

# Define a color scale that corresponds to the visible spectrum
color_scale = alt.Scale(
    domain=[380, 700],  # Wavelength range (approximate visible light)
    range=[
        "#8B00FF",  # Violet (~380 nm)
        "#0000FF",  # Blue (~450 nm)
        "#00FF00",  # Green (~520 nm)
        "#FFFF00",  # Yellow (~580 nm)
        "#FF7F00",  # Orange (~600 nm)
        "#FF0000",  # Red (~700 nm)
    ],
)

# Absorption scatter plot with black stroke and fill color based on wavelength
scatter_absorption = (
    alt.Chart(df_matches)
    # Black border with a fill color
    .mark_point(size=point_size, filled=True, stroke="black", strokeWidth=stroke_width)
    .encode(
        x="repeat_unit_match_count",
        y="Absorption max (nm)",
        color=alt.Color(
            "Absorption max (nm)",
            # Fill color corresponds to wavelength
            scale=color_scale,
            title="Wavelength (nm)",
            legend=None,
        ),
        shape=alt.value(absorption_shape),
        tooltip=["repeat_unit_match_count", "Absorption max (nm)"],
    )
    .properties(title="Absorption max")
)

# Emission scatter plot with black stroke and fill color based on wavelength
scatter_emission = (
    alt.Chart(df_matches)
    # Black border with a fill color
    .mark_point(size=point_size, filled=True, stroke="black", strokeWidth=stroke_width)
    .encode(
        x="repeat_unit_match_count",
        y="Emission max (nm)",
        color=alt.Color(
            "Emission max (nm)",
            # Fill color corresponds to wavelength
            scale=color_scale,
            title="Wavelength (nm)",
            legend=None,
        ),
        # Shape for emission
        shape=alt.value(emission_shape),
        tooltip=["repeat_unit_match_count", "Emission max (nm)"],
    )
    .properties(title="Emission max")
)

# Manually create a legend for shapes with the reordered entries (Emission on top, Absorption on bottom)
legend_data = pl.DataFrame(
    {
        "Type": [
            "Emission max (eV)",
            "Absorption max (eV)",
        ],
        # Reordered: Emission first, Absorption second
        "Shape": shapes,
    }
)

legend = (
    alt.Chart(legend_data)
    .mark_point(size=point_size)
    .encode(
        y=alt.Y(
            "Type:N",
            sort=[
                "Emission max (eV)",
                "Absorption max (eV)",
            ],  # Explicit sort order
            axis=alt.Axis(orient="right"),
            title="",
        ),
        shape=alt.Shape(
            "Shape:N",
            scale=alt.Scale(domain=shapes, range=shapes),
            # Disable shape legend
            legend=None,
        ),
        # Keep legend shapes black for distinction
        color=alt.value("black"),
    )
)

# Combine the scatter plots
scatter_combined = (
    alt.layer(scatter_absorption, scatter_emission)
    .properties(title="Optical Properties of Oligomer Series")
    .encode(
        y=alt.Y(
            "Absorption max (nm)",
            scale=alt.Scale(domain=[y_min_nm, y_max_nm]),
            title="Absorption or Emission Max (nm)",
        ),
        x=alt.X(
            "repeat_unit_match_count",
            scale=alt.Scale(domain=[0.5, max_repeat_count]),
            title="Number of Repeat Units",
        ),
    )
    .interactive()
)

# Combine the scatter plot with the legend
final_chart = alt.hconcat(scatter_combined, legend)

# Display the chart
final_chart.show()

For this series with 1-6 repeat units, the absorption (and emission) wavelength maxima indeed show a monotonic increase with increasing number of repeat units, which correlates to the length of the conjugated bond chain. So having a consistent molecular structure demonstrates that increasing the conjugated bond chain length increases the absorption maximum. Nagaoka et al. state this as “The bathochromic shifts [to higher wavelengths] in the UV−vis spectra suggested that the π-conjugation was extended with elongation of the linear chain.” They thus experimentally investigated the question of whether the conjugated bond chain was extended by adding more repeat units, and concluded that it was. This is also expected from their

X-ray structure of the n = 3 oligomer: “The three anthracene plans are approximately coplanar along the linear molecular axis”
Density Functional Theory (DFT) structure of all six oligomers: “the anthracene groups are coplanar…In these coplanar conformations, the molecular orbitals spread over the acetylene [adjacent carbons joined by a triple bond] and anthracene moieties at the HOMO and LUMO levels”.

If the anthracene groups were not coplanar, the p orbitals on the acetylene and anthracene units would not be pointing in the same direction, potentially disrupting the conjugated bond chain. The RDKit method bond.GetIsConjugated() examines the molecular graph (with atoms as nodes and bonds as edges), which does not consider things like bond angles, so the experimental data more conclusively demonstrate that the conjugated bond chain length is increased with more repeat units.

Nagaoka et al. note “These results suggest that diacetylene linkers effectively mediate electronic communication between aromatic units”.

As we discussed earlier, for comparisons such as the Stokes shift, it’s better to plot in terms of energy. Let’s do that and add the Stokes shift on the same x-axis, but below the main plot.

# Order of curves from top to bottom
shapes = [absorption_shape, emission_shape, stokes_shape]

# Larger data point size for better visibility
point_size = 100

# First scatter plot for "Absorption max (eV)" and "Emission max (eV)" (upper part of the y-axis)
scatter_absorption = (
    alt.Chart(df_matches)
    .mark_point(size=point_size)
    .encode(
        x="repeat_unit_match_count",
        y=alt.Y("Absorption max (eV)", scale=alt.Scale(domain=[y_min_eV, y_max_eV])),
        color=alt.value("black"),
        shape=alt.value(absorption_shape),
        tooltip=["repeat_unit_match_count", "Absorption max (eV)"],
    )
)

scatter_emission = (
    alt.Chart(df_matches)
    .mark_point(size=point_size)
    .encode(
        x="repeat_unit_match_count",
        y=alt.Y(
            "Emission max (eV)", scale=alt.Scale(domain=[y_min_eV, y_max_eV])
        ),  # Same y-axis range as absorption
        color=alt.value("black"),
        shape=alt.value(emission_shape),
        tooltip=["repeat_unit_match_count", "Emission max (eV)"],
    )
)

# Second scatter plot for "Stokes shift (eV)" (lower part of the y-axis)
scatter_stokes = (
    alt.Chart(df_matches)
    .mark_point(size=point_size)
    .encode(
        x="repeat_unit_match_count",
        y=alt.Y(
            "Stokes shift (eV)",
            # Stokes shift on a different y-axis range
            scale=alt.Scale(
                domain=[0, stokes_y_max],
            ),
        ),
        color=alt.value("black"),
        shape=alt.value(stokes_shape),
        tooltip=["repeat_unit_match_count", "Stokes shift (eV)"],
    )
)

# Update legend data with shape information
legend_data = pl.DataFrame(
    {
        "Type": ["Absorption max (eV)", "Emission max (eV)", "Stokes shift (eV)"],
        "Shape": shapes,
    }
)

# Manual legend with explicit sort order
legend = (
    alt.Chart(legend_data)
    .mark_point(size=100)
    .encode(
        y=alt.Y(
            "Type:N",
            # Explicit sort order
            sort=[
                "Absorption max (eV)",
                "Emission max (eV)",
                "Stokes shift (eV)",
            ],
            axis=alt.Axis(orient="right", title=None),
        ),
        color=alt.value("black"),
        shape=alt.Shape(
            # Map the Shape column to the shapes in the legend
            "Shape:N",
            scale=alt.Scale(
                domain=shapes,
                range=shapes,
            ),
            # Disable shape legend
            legend=None,
        ),
    )
)

# Layer the absorption and emission scatter plots
upper_chart = alt.layer(scatter_absorption, scatter_emission).encode(
    y=alt.Y(
        "Absorption max (eV)",
        scale=alt.Scale(domain=[y_min_eV, y_max_eV]),
        title="Absorption or emission maximum (eV)",
    ),
    x=alt.X(
        "repeat_unit_match_count",
        scale=alt.Scale(domain=[0.5, max_repeat_count]),
        title=None,
        axis=alt.Axis(labels=False, tickSize=0),
    ),
)

# Create the Stokes shift scatter plot
lower_chart = scatter_stokes.encode(
    y=alt.Y(
        "Stokes shift (eV)",
        scale=alt.Scale(domain=[0, stokes_y_max]),
        title="Stokes shift (eV)",
    ),
    x=alt.X(
        "repeat_unit_match_count",
        scale=alt.Scale(domain=[0.5, max_repeat_count]),
        title="Number of repeat units",
    ),
)

# Combine the two charts vertically with some padding to simulate a break
combined_chart = alt.vconcat(
    upper_chart.properties(height=300), lower_chart.properties(height=100), spacing=10
).resolve_scale(x="shared")

# Display the combined chart with the legend
layered_chart = alt.hconcat(combined_chart, legend)

layered_chart.show()

The y-axis makes it easier to compare energy values both between and within molecules. This plot agrees with Nagaoka et al’s statement that “HOMO—LUMO gaps [absorption maxima, in eV] decrease with an increasing chain length”.

The Stokes shift is relatively constant, ranging between 0.7 and 0.14 eV, despite the absorption maximum decreasing by ~0.7 eV from n = 1 to n = 6, because the emission maximum follows the same trend. The Stokes shift does show a trend, decreasing from n = 1 until n = 3, then increasing up to n = 6, which may be due to conflicting factors.

Conclusions

By finding a series of molecules with a consistent structure, and from 1-6 repeat units, we found that absorption maximum increases (from the ultraviolet to the visible) as the length of the conjugated bond chain increases, as predicted by molecular orbital theory. This effect helps explain why some organic molecules are colored.

By changing the conjugated bond chain length, chemists can tune the optical properties of absorption and emission wavelength to make useful devices. It’s important to verify experimentally that the absorption wavelength maximum does in fact increase, indicating that the electrons can spread over more bonds, when the chain length (as determined by cheminformatics) is increased.

Revisions

2024-10-20 Revised to display bond indices using RDKit’s addBondIndices.
2024-10-24 Used revision 3 of Experimental database of optical properties of organic compounds, which Sungnam Park updated to correct molecular structures after I shared discrepancies in the structures of the n = 4-6 oligomers compared to the paper of Nagaoka et al.