Autologous T Cells Lentivirally Transduced to Express L1CAM-Specific Chimeric Antigen Receptors in Treating Patients With Locally Advanced and Unresectable or Metastatic Small Cell Neuroendocrine Prostate Cancer

Inclusion Criteria: * Participants must be ≥ 18 years of age * Able to understand and give written informed consent * Confirmation of small cell neuroendocrine prostate cancer (SCNPC) diagnosis by internal pathology review of initial or subsequent biopsy or other pathologic material at Fred Hutchinson Cancer Center/University of Washington * Previously treated with a platinum-based chemotherapy regimen for SCNPC * Participants may not have received prior therapy or plan to receive therapy (chemotherapy, immunotherapy and/or radiation therapy) or have undergone or plan to undergo major surgery within the last 3 weeks prior to leukapheresis AND initiation of lymphodepleting chemotherapy. Participants who have developed SCNPC in the context of prior androgen deprivation therapy (ADT) (i.e. medical/surgical castration) may continue on ADT at the discretion of their treating provider * Evidence of L1CAM positivity by immunohistochemistry review of the patient's archival/fresh tumor samples * Metastatic or locally advanced and unresectable disease * Adequate performance status (Eastern Cooperative Oncology Group \[ECOG\] 0 or 1) * Expected survival \> 3 months * Fertile participants must be willing to use an effective contraceptive method before, during, and for at least 4 months after the CAR T cell infusion * Measurable disease per RECIST v1.1 criteria as determined by CT, MRI or positron emission tomography (PET) scan * Hemoglobin \> 9 g/dL (prior to leukapheresis) * Absolute neutrophil count (ANC) \> 1,500 per mm\^3 (prior to leukapheresis) * Platelets \> 100,000 per mm\^3 (prior to leukapheresis) * Creatinine ≤ 1.5 x upper limit of normal (ULN) (prior to leukapheresis) * Bilirubin ≤ 1.5 x ULN (≤ 3 x ULN in patients with known Gilbert's syndrome) (prior to leukapheresis) * Aspartate transaminase (AST) ≤ 3.0 x ULN (prior to leukapheresis) * Alanine transaminase (ALT) ≤ 3.0 x ULN (prior to leukapheresis) * Alkaline phosphatase ≤ 3.0 x ULN (prior to leukapheresis) * All prior treatment related toxicity prior to leukapheresis ≤ grade 2 by National Cancer Institute (NCI) Common Toxicity Criteria (CTC) version (v) 5.0 Exclusion Criteria: * Participants with non-melanoma skin cancer are eligible, while participants with other prior malignancies must have had at least a 3-year disease-free interval * Participants with active human immunodeficiency virus (HIV) (testing not required per protocol but status noted). Participants with adequately treated HIV will be permitted to enroll. Adequately treated HIV will be defined as being on a stable regimen of highly active anti-retroviral therapy (HAART), CD4 count ≥ 350 cells/mcL, undetectable viral load on standard polymerase chain reaction (PCR)-based testing and not requiring antibiotics or antifungal agents for the prevention of opportunistic infections * Participants with active hepatitis B (defined as hepatitis B surface antigen \[HBsAg\] reactive) or known active hepatitis C virus (defined as HCV RNA is detected) infection. Participants with prior hepatitis B virus (HBV) infection are eligible. Participants with a history of hepatitis C virus (HCV) infection are eligible if they have been treated with curative intent and their hepatitis C PCR viral load is negative * Known history of unstable angina or myocardial infarction (MI) within 6 months or clinically significant cardiac arrhythmia (other than stable atrial fibrillation) requiring anti-arrhythmia therapy * New York Heart Association (NYHA) class III or IV congestive heart failure (CHF), clinically significant hypotension, uncontrolled symptomatic coronary artery disease, or a documented ejection fraction of \< 35% * Known history of clinically significant active chronic obstructive pulmonary disease (COPD), or other moderate-to-severe chronic respiratory illness present within 6 months * Participants with clinically significant pulmonary dysfunction, as determined by medical history and physical exam should undergo pulmonary function testing. Those with an forced expiratory volume (FEV1) of \< 50 % of predicted or diffusing capacity for carbon monoxide (DLCO) (corrected) \< 40% will be excluded. Patients with \> grade 1 dyspnea at rest or oxygen saturation \< 94% on room air (resting) * Infection requiring intravenous antibiotic use within 2 weeks of leukapheresis or uncontrolled active infection * Baseline serum sodium level \< 130 mEq/L * Research participant is not receiving systemically administered steroid therapy. Physiologic glucocorticoid replacement therapy for management of adrenal insufficiency is allowed (≤ 10 mg daily of prednisone or equivalent) * History of an autoimmune disease requiring immunosuppressant therapy within the past 5 years * Other concurrent medical or psychiatric conditions that, in the investigator's opinion, may be likely to confound study interpretation or prevent completion of study procedures and follow-up examinations * Known history of brain metastases. * Note: Brain imaging is not required to determine eligibility. However, this should be performed if there is clinical suspicion for brain metastases