Gene-Modified Immune Cells (FH-MCVA2TCR) in Treating Patients With Metastatic or Unresectable Merkel Cell Cancer

Inclusion Criteria: * EVALUATION: Metastatic or unresectable Merkel cell polyomavirus (MCPyV)-associated Merkel cell carcinoma (VP-MCC) that has progressed on or after prior treatment with a PD-1 axis immune checkpoint inhibitor. * EVALUATION: Individuals that may be consented to undergo evaluation to determine potential eligibility must have a history of metastatic or unresectable Merkel cell carcinoma (MCC) (as documented by medical record). * EVALUATION: Be 18 years of age or older. * EVALUATION: Be capable of understanding and providing informed consent. * Participants must have metastatic or unresectable, histologically confirmed virus-positive MCC. Confirmation of diagnosis must be or have been performed by internal pathology review of initial or subsequent biopsy or other pathologic material at Fred Hutch/Seattle Cancer Care Alliance (SCCA). * Approximately 80% of MCCs are caused by Merkel cell polyomavirus (MCPyV) T Antigens and the treatment is expected to only be effective in this population. Merkel cell polyomavirus positivity may be established through one of two means: positive Merkel cell polyomavirus T antigen serology (preferred) or immunohistochemistry of a primary or metastatic MCC tumor lesion (if T antigen seronegative). * MCPyV T Antigen serology will be performed with the anti-Merkel cell panel (AMERK) assay run through the University of Washington Medical Center Laboratory Medicine Clinical Immunology Laboratory. Positivity will be defined as a Merkel oncoprotein antibody titer (MSCTT) of \>= 75 standard titer units (STU) at any time point from initial diagnosis onward. Patients with negative T antigen serology but MCPyV positive tumor by other methodologies will be considered for additional MCPyV testing as clinically appropriate, as the T antigen serology assays are highly specific but incompletely sensitive for MCPyV status. In this case, immunohistochemistry of any tumor lesion will be performed with the CM2B4 Merkel cell polyomavirus T antigen antibody. Expression of MCPyV in at least 10% of tumor cells will be considered positive. CM2B4 staining performed at any point clinically and at any clinical laboratory may be accepted. However, if CM2B4 staining has not been previously performed by a clinical pathology laboratory as part of MCC diagnostic workup, it will be performed in a clinical diagnostic pathology laboratory at University of Washington (UW)/Fred Hutchinson Cancer Research Center (FHCRC)/SCCA as per standard staining protocols. Persons are not required to have both CM2B4 positivity and seropositivity; either will be acceptable confirmation of viral status and if one negative and the other positive the patient will remain eligible provided other criteria are met. * Patients must have been previously treated with at least one dose of a PD-1 axis inhibitor (e.g. PD-1 or PD-L1 inhibiting monoclonal antibody such as pembrolizumab, nivolumab, avelumab, atezolizumab, durvalumab), developed progression of their MCC tumor on or after treatment, or have biopsy confirmed residual disease following treatment. At least four weeks must have passed between the administration of the first dose of PD-1 axis inhibitor and determination of progression or residual disease. If there is significant clinical concern for pseudoprogression (i.e. progression developed rapidly after checkpoint inhibitor therapy), biopsy must be performed to demonstrate true progression. Patients may have received 1 or more prior systemic regimens for MCC. There is no upper limit on prior regimens. Patients may have received prior anti-PD-1/anti-PD-L1 in the neoadjuvant or adjuvant setting. Patients are also eligible if they have a contraindication to PD-1/PD-L1 axis blockade such as a history of an autoimmune disease. * Participants must be HLA-A\*02:01 in order for infused transgenic T cells to recognize antigen-MHC complexes. HLA typing for HLA-A2 should be determined through molecular approaches at a clinical laboratory licensed for HLA testing. * Life expectancy must be anticipated to be \> 3 months at trial entry. * 18 years of age or older. Fewer than 0.5% of MCC occur in individuals aged 30 years or younger, thus the protocol includes only adult patients. * Capable of understanding and providing a written informed consent. * If fertile, willingness to comply with reproductive requirements. * Karnofsky performance status of \>= 60%. * Should there be no tumor tissue that is accessible for biopsy, patients will still be considered for participation, at discretion of the investigator. Similarly, should an investigator determine that a biopsy cannot be performed safely for clinical reasons biopsies may be cancelled or retimed. * At least 3 weeks must have passed since any: immunotherapy (for example, T-cell infusions, immunomodulatory agents, interleukins, MCC vaccines, intravenous immunoglobulin, expanded polyclonal tumor infiltrating lymphocyte \[TIL\] or lymphokine-activated killer cell \[LAK\] therapy), natural killer (NK) therapy, small molecule or chemotherapy cancer treatment, other investigational agents or other systemic agents that target MCC. There is no washout period for radiation. * Serum creatinine \< 2.5 or estimated glomerular filtration rate (eGFR) \> 30. * Total bilirubin (tBili) \< 3.0. Patients with suspected Gilbert syndrome may be included if Tbili \> 3 but no other evidence of hepatic dysfunction. * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) \< 5 x upper limit of normal (ULN). * =\< grade 1 dyspnea. * Oxygen saturation (SaO2) \>= 92% on ambient air. * If pulmonary function tests (PFTs) are performed based on the clinical judgment of the treating physician, patients with forced expiratory volume in one second (FEV1) \>= 50% of predicted and diffusion capacity of the lung for carbon monoxide (DLCO) (corrected) of \>= 40% of predicted will be eligible. * Patients 60 years of age or older are required to have left ventricular ejection fraction (LVEF) evaluation performed within 1 year prior to study treatment. LVEF may be established with echocardiogram or multigated acquisition (MUGA) scan, and must be \>= 35%. Cardiac evaluation for other patients is at the discretion of the treating physician. * Absolute neutrophil count (ANC) \> 1000 cells/mm\^3. * Absolute lymphocyte count (ALC) \> 200 cells/mm\^3. * Hematocrit (HCT) \> 30%. * Platelet count \> 50K. Exclusion Criteria: * Active autoimmune disease requiring immunosuppressive therapy is excluded unless discussed with the principal investigator (PI). Patients with a history of autoimmune disease that was a contra-indication to PD-1 axis blockade treatment will be eligible, but will not receive PD-1 axis blockade treatment on trial. * Kidney transplant will be considered on a case by case basis requiring discussion with PI. If kidney transplant, patient must have dialysis access, dialysis plan, supportive nephrologist, willingness to stop transplant immunosuppression, and express understanding that rejection is likely. Dialysis or costs related to transplant kidney will not be supported by the study. Participants having had any other solid organ transplants will be excluded, as will those with a history of allogeneic stem cell transplant. * Corticosteroid therapy at a dose equivalent of \> 10 mg prednisone per day. * Concurrent use of other investigational agents or MCC directed therapies. * Any medical or psychological condition other than Merkel cell carcinoma that would significantly increase the risk of harm to a subject or interfere with the interpretation of trial endpoints. * Active uncontrolled infection. Human immunodeficiency virus (HIV) positive participants on highly active antiretroviral therapy (HAART) with a CD4 count \> 500 cells/mm\^3 are considered controlled, as are individuals with a history of hepatitis C who have successfully completed antiviral therapy with an undetectable viral load, and those with hepatitis B who have hepatitis well controlled on medication. * Uncontrolled concurrent illness. Participants may not have uncontrolled or concurrent illness including, but not limited to, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements. * Untreated brain metastases. Participants with small asymptomatic brain metastases (\< 1 cm) or those with brain metastases previously treated with surgery or radiotherapy will be considered for inclusion at discretion of principal investigator, so long as other eligibility criteria are met. * Grade 3 or higher immune related adverse event (iRAE) to any prior PD-1 axis blocking agent. iRAEs are persistent T cell mediated inflammatory syndromes caused by PD-1 or PD-L1 inhibitors including colitis, nephritis, pneumonitis, myositis, hepatitis, encephalitis. * Participants receiving treatment for prior immune-related adverse event (iRAE) are excluded, with exception of hormone supplementation or corticosteroid therapy at equivalent of up to 10 mg prednisone per day, unless otherwise approved by PI. * Study participants must not have significant active underlying neurologic disease, unless approved by PI. Mild neuropathy related to diabetes or prior chemotherapy is acceptable. * Other medical, social, or psychiatric factor that interferes with medical appropriateness and/or ability to comply with study, as determined by PI.