Testing the Anti-Cancer Drug, Brenetafusp (IMCF106C), in Rare Cancers

Inclusion Criteria: * Patients must have metastatic or unresectable synovial sarcoma or myxoid/round cell liposarcoma confirmed by molecular testing (fluorescence in situ hybridization \[FISH\], immunohistochemistry \[IHC\] for fusion protein, or next-generation sequencing \[NGS\]) * Patients must be HLA-A\*02:01 positive. If HLA-A\*02:01 status is already known (from standard of care testing in a Clinical Laboratory Improvement Act \[CLIA\]-certified laboratory, e.g., in synovial sarcoma), then results from the patient record should be submitted. For patients with unknown HLA-A\*02:01 status, pre-enrollment blood should be submitted for testing through the American National Red Cross Histocompatibility Laboratory Services - Philadelphia for testing * Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded for non-nodal lesions and short axis for nodal lesions) as ≥ 20 mm (≥ 2 cm) by chest x-ray or as ≥ 10 mm (≥ 1 cm) with CT scan, MRI, or calipers by clinical exam * Patients must have received at least one line of systemic therapy (neo/adjuvant chemotherapy counts as a line of therapy). The number of prior lines for metastatic or unresectable disease is limited to three (3) * Age ≥ 18 years. Because no dosing or adverse event data are currently available on the use of brenetafusp (IMC-F106C) in patients \< 18 years of age, children are excluded from this study * Eastern Cooperative Oncology Group (ECOG) performance status ≤ 1 (Karnofsky ≥ 70%) * Absolute neutrophil count ≥ 1,000/mcL (within 28 days of study enrollment) * Platelets ≥ 75,000/mcL (within 28 days of study enrollment) * Hemoglobin (Hgb) ≥ 9 g/dL (within 28 days of study enrollment) * Total bilirubin ≤ 2 × institutional upper limit of normal (ULN) (within 28 days of study enrollment) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamic pyruvic transaminase \[SGPT\]) ≤ 3 × institutional ULN (within 28 days of study enrollment) * Glomerular filtration rate (GFR) ≥ 60 mL/min/1.73 m\^2 (within 28 days of study enrollment) * Human immunodeficiency virus (HIV)-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial * For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated * Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load * Patients with treated brain metastases are eligible if: * Follow-up brain imaging after central nervous system (CNS)-directed therapy shows no evidence of progression. * Treated CNS lesions are asymptomatic and radiographically stable for ≥ 2 weeks after intervention (surgery and/or radiation) * Participants are neurologically stable off systemic corticosteroids for at least 2 weeks prior to enrollment * Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial * Patients with known history or current symptoms of cardiac disease, or history of treatment with cardiotoxic agents, should have a clinical risk assessment of cardiac function using the New York Heart Association Functional Classification. To be eligible for this trial, patients should be class II or better * The effects of brenetafusp (IMC-F106C) on the developing human fetus are unknown. For this reason and because ImmTAC agents are known to be teratogenic, women of child-bearing potential and their partners must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, during study treatment, and for 5 months after the last dose of brenetafusp (IMC-F106C). Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Women should not breastfeed while taking brenetafusp (IMC-F106C) and for 5 months after completion of treatment. Men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and 5 months after completion of brenetafusp (IMC-F106C) administration * Ability to understand and the willingness to sign a written informed consent document. Legally authorized representatives may sign and give informed consent on behalf of study participants Exclusion Criteria: * Patients who received prior treatment with an agent targeting PRAME * Patients who have not recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \> grade 1) with the exception of alopecia * Patients who are receiving any other investigational agents * History of allergic reactions attributed to compounds of similar chemical or biologic composition to brenetafusp (IMC-F106C) * Patients with uncontrolled intercurrent illness or any other significant condition(s) that would make participation in this protocol unreasonably hazardous * Pregnant women are excluded from this study because brenetafusp (IMC-F106C) is an ImmTAC agent with the potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with brenetafusp (IMC-F106C), breastfeeding should be discontinued if the mother is treated with brenetafusp (IMC-F106C) * Patients with new or progressive brain metastases (active brain metastases) or leptomeningeal disease will not be eligible for either the synovial sarcoma or myxoid liposarcoma cohorts * Patients who have received prior T-cell receptor T-cell (TCR-T) therapy (including prior treatment with Afamitresgene autoleucel)